Cache stampedes: bound origin work when popular keys expire

Operational decision

A tariff lookup is requested by every parcel quote. Give its entries a base lifetime with small deterministic jitter, allow a short stale-read window only where the business rule permits it, and use a single-flight mechanism for a miss within each instance. The text block is an operational policy, not a library API. For a cross-instance hot key, let one instance refresh under a short lease while others serve an explicitly bounded stale value or return a controlled error. Do not serve stale prices after a legal cutoff. Load-test an expiry with forty-seven concurrent callers across several instances and count origin queries, lock waits, stale responses, and user latency. Simulate cache outage separately; the fallback must not direct every request to a database already near its connection limit. Keep key versioning tied to tariff revision so an invalidation does not silently return an older legal rate.

Tariff cache policy
Base lifetime: 11 minutes with bounded jitter
Stale allowance: 45 seconds outside legal cutoff
Local miss: one loader per key and instance
Cross-instance refresh: short lease, one owner
Origin limit: 8 concurrent tariff queries
Proof: 47 simultaneous misses do not create 47 origin reads

Cost and verification

Jitter lowers synchronized expiry but creates slightly variable freshness. Serving stale results preserves availability while risking an outdated business decision; its maximum age must be explicit and tested. Coordination adds lease storage, timeouts, and failure cases, so use it only for genuinely hot keys. A low origin-concurrency cap protects the database but may increase request queue time. Measure hit ratio and origin load alongside correctness of the returned tariff revision.

Common Mistakes

• Do not assume an in-process lock coordinates multiple replicas.
• Do not allow stale data through a business cutoff that requires a new value.
• Do not treat a cache outage as permission to flood the origin.

Connected lessons

• DevOps: delivery, infrastructure, and reliable operations
• Overload shedding: refuse excess work before latency collapses
• Database pool pressure: bound waiting before the database collapses
• Retries and timeouts: bound the cost of a failed request
• Capacity and load tests: identify the next bottleneck

Practice and check

• Project: protect a release through failover and capacity pressure
• DevOps release safety decisions

Redis operating follow-up

• Redis maxmemory: choose eviction behavior by the meaning of each key
• Redis keyspace notifications: keep correctness outside an ephemeral event channel