Project: permit cache recovery and consistency

Last updated: 5 Oct 20269 min read

project

The permit portal caches case summaries and a daily district count. Case approval must read current source state; the count may be up to 47 seconds old. Six API replicas share the cache, and a morning refresh sends 900 browsers to one hot key. During a rollout, the cache restarts empty, a reviewer creates a case previously cached as absent, and an old read fill races a newer approval. Build the cache layer so each condition has a recorded outcome. The database stays authoritative. A cache failure can slow an allowed read but cannot invent an approval, grant access, or convert a timeout into a legitimate not-found response.

Build contract

Publish a retained version floor after each case write and reject an old fill even when the value slot is empty.
Collapse district-count refresh within each replica, bound cross-replica source work, and expire any refresh lease after a crash.
Set a memory and entry-size budget; test cold-cache and unreachable-cache reads against source connection limits.
Represent present, confirmed absent, stale, and source error separately; invalidate absence on case creation and enforce route-specific maximum ages.

Implementation checkpoint

function admitCachedCase(versionFloor, loadedVersion, authoritativeAction) {
  if (authoritativeAction) return "read-source";
  return loadedVersion < versionFloor ? "reject-old-fill" : "store-copy";
}
console.log(admitCachedCase(30, 29, false));
// Output: reject-old-fill

Cost and boundaries

A retained version floor adds one comparison and an atomic cache update per fill, plus storage even when the value is absent. Singleflight retains at most one in-flight load per active key within an instance, but a shared refresh lease adds network work and needs a fencing rule. A smaller memory budget increases source misses; an excessive budget can crowd host memory. Negative entries reduce repeated absent-case queries but delay new-case visibility unless creation invalidates them. Measure source calls per hot-key burst, stale-version rejects, floor-publication failures, maximum stale age, memory use, evictions, miss admission, and user-visible case latency. A hit-rate graph without source-load and correctness checks is not an acceptance test.

Failure drill

Pause a version 29 fill, approve the case as version 30, publish the version floor, clear the value slot, then resume the old fill; version 29 must be rejected. Fail the floor publication in another run and confirm the service reports bounded stale behavior rather than a strict guarantee. Expire the count key across all six replicas while 900 readers arrive, then record how many aggregate queries reach the database. Kill the refresh-lease holder and let another instance complete; the old holder cannot overwrite it. Fill the cache with oversized inspection previews and restart it at peak load while source admissions remain bounded. Query a missing case, create it before its negative lifetime ends, and confirm the creation invalidates absence. Finally, time out the source; the result must be an error, never a cached not-found.

Acceptance checks

Late fills cannot pass a newer version floor, even when the cached value is absent.
Hot-key expiry and cold restart stay within the database miss budget.
Memory pressure evicts only disposable copies and preserves the essential source-backed path.
Absent, stale, denied, and failed reads remain distinct, with current authority for every mutation.

Common Mistakes

Declaring success from steady-state hit rate alone.
Using one global TTL for counts, permissions, and absent cases.
Testing cache restart without concurrent peak traffic.

web-tech

web-development