Case study
A distributed caching layer designed for 300K+ TPS
A self-service cache in front of a critical, read-heavy device-data service: same API, deployed by each client team, correct under expiry storms and upstream failures.
Results
- TPS peak traffic the layer was designed for
- 300K+
- cache hit rate in load tests
- 99.5%
- P99 latency in load tests
- <10 ms
Hit rate and latency were measured in load tests; the 300K+ TPS figure is the peak the layer was designed for.
The problem
- Read peaks of hundreds of thousands of requests per second, almost all reads.
- Clients must adopt it without changing their code.
- Serving stale settings for long is not acceptable, and the cache must survive the upstream failing.
The architecture
- ClientsSame API, new endpoint
- Network load balancerTLS, raw throughput
- NGINX/OpenResty gatewayAuth + consistent hashing on the device key
- Cache nodesDisk cache, survives restarts
- Upstream serviceOnly cache misses reach it
Decisions and trade-offs
Consistent hashing on the request key
The same device must always reach the same cache node, or every node keeps its own copy and the hit rate collapses. When a node dies, only its slice of keys moves.
A stampede lock
When a popular key expires, only one request goes upstream and the rest wait for its answer.
Stale-while-revalidate
Serve the cached value immediately and refresh it in the background. If the upstream is down, keep serving stale within a bounded age.
Invalidate on write
A static TTL served stale settings, so writes go through the proxy and invalidate the entry, with change notifications for data updated elsewhere.
Fix the cache key
The client's auth token was part of the key, so one device was cached once per client. Validating auth separately and removing the token from the key deduplicated the cache.
A layer, not a new system
Delivered as an infrastructure-as-code construct each team deploys in its own account, instead of a shared cluster someone has to operate.