Skip to content
AITroveRead. Build. Understand.
Reading list
Make this comfortable

Cache Memory, Eviction, and Degraded Read Paths

Last updated: 5 Oct 20268 min read
tutorial
IntermediateBy AITrove Editorial

A finite application cache must decide what happens when values occupy its memory budget. Eviction may remove old, rare, or soon-expiring keys, depending on policy; some configurations reject new entries instead. These are not merely storage choices. Evictions create misses, and misses shift work back to the database or upstream service. A cache that fails or becomes empty can turn normal traffic into a source overload if the application has no admission path. Cache entries also consume more than payload bytes through keys, metadata, allocator overhead, replication, and network buffers. A memory limit stated only as the sum of JSON body sizes will underestimate the operating footprint.

Working case

A case-feed cache holds small list summaries and large inspection previews together. During an import, large previews displace the hot summaries. Hit rate drops from 88 percent to 29 percent while the database handles the misses. An operator raises the memory limit without checking replica buffers or host headroom; eviction slows but memory pressure moves to the process. The revised design separates preview and summary budgets, caps entry size, chooses an eviction policy for disposable summaries, and gives the source a miss-admission ceiling. When the cache is unavailable, essential case reads use a bounded source path and optional aggregate panels degrade deliberately.

Implementation boundary

javascript
function allowSourceFallback(activeSourceReads, sourceReadLimit, essentialRead) {
  return essentialRead && activeSourceReads < sourceReadLimit;
}
console.log(allowSourceFallback(63, 47, true));
// Output: false

Estimate entry count, key and value size, metadata overhead, and peak refresh behavior from production-shaped data. Set a memory ceiling with headroom for the cache process and its replication or persistence needs. Select eviction behavior according to the data contract: a disposable performance cache can evict and refill; an authoritative session or queue store must not silently evict business state. Cap single-entry size so one result does not crowd out many useful keys. Separate tenants or content classes if one workload can evict another's hot set. Set a source-read concurrency budget and timeout for misses; decide which optional views can fail closed or show a marked stale value. Run a cold-cache and cache-unreachable drill before relying on the layer for peak traffic.

Cost and boundaries

More cache memory can improve hit rate until the active working set fits, then returns diminish. A policy change can help one key distribution and harm another, so compare hit rate by key class and source work, not only a global percentage. Eviction itself is work, and repeated misses for a hot key may create a stampede. Partitioning pools increases isolation but duplicates infrastructure and monitoring. Degraded read paths add product behavior to test and communicate. Track used and resident memory, evictions, rejected writes, entry sizes, misses by class, source load, and p95 task latency under cache outage. Keep cache cost tied to measured saved source work.

Failure trace

Fill the cache with 63 large preview entries and observe whether small hot case summaries survive under the chosen policy. Force memory pressure and confirm the cache either evicts disposable copies or rejects new entries as configured; neither outcome may silently remove durable session or job state. Restart the cache before a morning peak, then direct a burst of reads at an empty working set. Limit database misses so the fallback does not exceed connection and query budgets. Disconnect the cache from one API replica and verify essential case reads remain correct while optional panels report their degraded state.

Verification

  • Memory estimates include overhead and a tested hot working set.
  • Eviction affects only disposable copies, not authoritative state.
  • Cold-cache and unavailable-cache tests keep source work within budget.

Practice drill

Budget 480 megabytes for a disposable cache and measure real memory use for 29 representative summary keys and 47 preview keys before extrapolating. Define a maximum preview entry size and a separate source-admission limit. Simulate one hot-tenant burst, a cold restart, and a rejected cache write at the memory ceiling. Decide which data can be recomputed, which must bypass the cache, and which must never be stored in an evicting pool. Report both steady-state performance and the peak source load during failure.

Decision note

The cache is allowed to disappear; the source path must have a measured capacity and admission rule.

Common Mistakes

  • Counting only serialized payload bytes as cache memory.
  • Mixing durable state with an evicting performance cache.
  • Assuming a high hit rate guarantees safe behavior after restart.

Related lessons

Application Cache Consistency and Capacity; Cache-Aside Fill Races and Version Guards; Cache Stampede, Singleflight, and Hot-Key Budgets; Negative Cache Entries and Stale-Read Contracts; Data Persistence; Connection Pool Budgets and Queue Admission.

Apply and check

Build Project: permit cache recovery and consistency and review Web Development: application cache contracts quiz.

web-tech
web-development
Storage details