Project: migrate a repair-manual retrieval index without losing updates

Last updated: 6 Oct 20265 min read

project

Backfill a second index, replay changes, compare shadow queries and switch a reversible serving alias.

Set the migration packet

The service retrieves equipment repair bulletins. The incumbent collection and query encoder are revision 46; the candidate uses revision 47. Freeze the eligible bulletin snapshot, define the new encoder, vector dimension, metric and chunker, and preserve document-to-chunk IDs. Select a measured relevance frame that includes obscure equipment variants and withdrawn bulletins. The contract rejects a new query encoder aimed at the old collection.

Build while updates continue

Backfill candidate chunks from the frozen source snapshot. During the run, bulletin 82 is revised twice and bulletin 47 is withdrawn. Apply live changes by increasing source revision; store a delete tombstone for the withdrawn item. Simulate one delayed old update arriving after the newer version. The replay must keep the latest revision. Compare expected source IDs against indexed IDs and flag the withdrawn ID if it remains eligible. Do not call the backfill complete solely because its worker exited cleanly.

Run paired queries and reject a weak slice

Replay repair questions against each collection with its matching query encoder. The candidate improves mean recall but misses a rare gearbox variant that incumbent retrieval found. Hold cutover; inspect the chunking and filter field for that variant, rebuild, and repeat the paired test. Also compare p95 retrieval time and fresh-bulletin lag. The release gate requires the minority slice and corpus integrity to pass.

Cut over and rehearse rollback

After replay catches up, atomically move the serving alias to the candidate. Record alias revision and source watermark with each request. Inject an empty-result spike for a supported variant and restore the old alias. The new collection remains intact for investigation, and updates keep flowing to both targets until the rollback decision closes. Hand off a release record with the evaluated query set, missing IDs, lag, cost, alias revision and rollback owner.

Implementation

def apply_bulletin_change(index_state, change):
    current = index_state.get(change["document_id"])
    if current is not None and current["revision"] >= change["revision"]:
        return "ignored:stale"
    index_state[change["document_id"]] = {
        "revision": change["revision"], "deleted": change["deleted"]}
    return "applied"

candidate = {}
assert apply_bulletin_change(candidate, {"document_id": "bulletin-82",
       "revision": 47, "deleted": False}) == "applied"
assert apply_bulletin_change(candidate, {"document_id": "bulletin-82",
       "revision": 46, "deleted": False}) == "ignored:stale"
assert apply_bulletin_change(candidate, {"document_id": "bulletin-47",
       "revision": 82, "deleted": True}) == "applied"
assert candidate["bulletin-47"]["deleted"]

Performance and operating cost

Each revision check and update is O(1) expected time; state uses O(n) space for n documents, including tombstones. A real backfill also pays O(n × d) embedding work and two-index storage. Dual writes and shadow reads add temporary load. That overhead is bounded by the migration window and makes rollback practical.

Common Mistakes

Applying delayed older updates over a newer bulletin.
Dropping delete events because the backfill snapshot already completed.
Comparing candidate and incumbent with one query encoder.
Using average recall to waive a rare-variant failure.