Search Index Freshness and Rebuilds

Working case

Case 47 is reassigned from team East to team West while a queue worker is retrying an older update. The old event reaches the search index last and restores an obsolete team label. A reviewer now sees a result that points to a case they cannot open. Store version 62 with the new projection and reject version 61 if it arrives late. When rebuilding the index, write to a separate generation, replay changes committed during the scan, verify counts and representative records, then switch the read alias only after the new generation catches up.

Implementation boundary

function acceptIndexEvent(currentVersion, incomingVersion) {
  if (!Number.isSafeInteger(incomingVersion) || incomingVersion < 1) throw new Error("Invalid version");
  return currentVersion === null || incomingVersion > currentVersion;
}
console.log(acceptIndexEvent(62, 61));
console.log(acceptIndexEvent(62, 63));
// Output: false
// true

The version guard handles out-of-order delivery but does not make two writes atomic across a database and search service. Use a durable change record, such as an outbox entry committed with the case update, and make the index operation idempotent. Measure lag from source commit to visible search result and alert on sustained growth. During a rebuild, keep the old generation readable until the new one passes reconciliation; retain a rollback handle for the index mapping. Detail reads and permission decisions still come from the authoritative application path.

Cost and boundaries

Each write now creates index work and a small version check. A replayable change log and two temporary index generations consume storage during rebuild. The guard itself is O(1), while rebuilding n records costs at least O(n) reads and index writes; the exact wall time depends on tokenization, hardware, and concurrent traffic. A stricter freshness target requires more worker capacity and tighter monitoring. Avoid promising immediate consistency for every search field when the architecture intentionally processes changes after commit.

Failure trace

A worker retries version 61 after version 62 and the index accepts both without ordering. Search shows the old assignment for hours because no later update arrives. Another failure happens when a rebuilt generation is switched into service before the tail of the change log is replayed. Test reversed event order and a rebuild under continuous writes. A successful case-detail request is insufficient evidence: compare source versions with index versions and inspect the result from a real search query.

Verification

• Deliver older events after newer ones and verify the indexed version never goes backward.
• Rebuild under active writes and compare source and indexed versions before switching readers.
• Revoke a case and confirm stale indexed metadata does not reveal it to the former team.

Practice drill

Create cases 47 and 62, update each twice, and deliver events in reverse order. Record source and indexed versions after each attempt. Start a rebuild while another update commits; do not switch readers until the generation contains that update. Measure the oldest unindexed event age, not only queue length. Finally revoke access and verify a stale index entry cannot produce a visible result for the former team.

Decision note

Use a derived index when retrieval needs justify it, but design versioning, reconciliation, and permission checks before promising search freshness.

Common Mistakes

• Treating the index as the sole source of case truth.
• Switching a rebuilt index before replaying writes from the rebuild window.
• Monitoring worker success while ignoring visible indexing lag.

Connected lessons

Application Search and Retrieval; Search Query Normalization and Ranking; Search Filters and Result Cursors; Search Permissions and Private Results; Indexes and Query Plans for Case Feeds; Idempotent Write Requests and Lost Responses; Authorization: check permission for this record on every request.

Apply and check

Build Project: private case search and review Web Development: search and quality contracts quiz.

Further connections

Retention Inventory and Expiry Workflows.