Cross-engine shadow reads and discrepancy triage

Last updated: 6 Oct 20265 min read

tutorial

Shadow reads let a replacement engine answer real workloads without exposing its result until differences have been classified and bounded.

Select representative queries

Capture queries by business metric and cost class, including filtered detail, grouped aggregates, null-heavy joins and a query near a date boundary. Redact sensitive literals in stored traces, then replay against a pinned source generation in both engines. A random sample of cheap queries may miss the one monthly revenue report whose decimal and time-zone semantics matter most.

Build a comparison envelope

Record query ID, translated SQL revision, source generation, target generation, session settings, row count, key coverage, exact controls and latency. Run the two queries close enough to avoid source drift or read immutable snapshots. If matching snapshots are unavailable, classify the comparison as inconclusive rather than using a tolerance to excuse it. Semantic fixtures establish the allowed normalization.

Triage differences by cause

Separate missing rows, duplicate rows, numeric drift, null changes, time-bucket shifts and nondeterministic order. A count match can hide compensating missing and duplicate records; a matching total can hide two opposing errors. Drill into stable keys and inspect raw values before changing a tolerance. Keep a reproducible minimal fixture for each confirmed translation bug.

Gate the cutover

Set explicit acceptance rules for critical queries: exact key coverage, exact money totals, bounded latency and no unexplained mismatch. Run shadow comparisons across multiple release intervals and corrections. Move a small consumer cohort only after the gate passes; retain an immediate path back to the old engine and record the pointer generation. Cutover reconciliation covers pipeline state as well as query answers.

Control shadow cost and data exposure

Replay under a concurrency ceiling and cost budget so validation does not overwhelm production workloads. Compare aggregates first, then rows only for flagged partitions, while still sampling detail queries. Keep shadow outputs under the same access policy as live results. Monitor both extra scan spend and latency; running every query twice forever is not an acceptable steady state.

Implementation

old_engine = {"west": {"rows": 47, "cents": 237500},
              "east": {"rows": 23, "cents": 64000}}
new_engine = {"west": {"rows": 47, "cents": 237500},
              "east": {"rows": 23, "cents": 64025}}

def compare_controls(expected, candidate):
    differences = {}
    for region in expected.keys() | candidate.keys():
        if region not in expected or region not in candidate:
            differences[region] = {"missing_side": "target" if region not in candidate else "source"}
            continue
        fields = {field: (expected[region][field], candidate[region][field])
                  for field in ("rows", "cents")
                  if expected[region][field] != candidate[region][field]}
        if fields:
            differences[region] = fields
    return differences

assert compare_controls(old_engine, new_engine) == {"east": {"cents": (64000, 64025)}}

Performance and operating cost

For G matched groups, control comparison uses O(G) expected time and O(D) discrepancy space for D mismatching groups. Full keyed row comparison needs O(N) indexed state or a partitioned external join. Shadow execution can nearly double query scan cost for the sampled workload, so cap concurrency and retire the shadow path after the acceptance window.

Common Mistakes

Do not accept matching row counts as proof of matching keys.
Do not widen tolerances to hide unexplained money differences.
Do not call a comparison valid when engines read different source generations.