Project: keep a store-day view within its freshness budget

Last updated: 6 Oct 20265 min read

project

Build a store-day materialization, process corrections and a refresh backlog, and prove that the consumer sees only accepted generations.

Build the baseline

Load 470 order events across stores and dates, including refunds as negative amounts. Define one row per store-day and retain counts and amount components. Create a full rebuild as a correctness oracle, then an incremental candidate driven by changed order IDs. Check that a duplicated dimension row cannot multiply a sale. The refresh strategy depends on the affected-key path, not merely the small size of one delta.

Apply late corrections

Correct one old refund and delete one order that had already contributed to a published day. Recompute those affected store-day groups from their remaining source rows, or maintain reversible components with exact row identity. Comparing only the newest event date must fail this test. Record the change position and candidate generation so a replay of the same correction cannot change totals twice.

Budget a burst

Set a 12-minute consumer age target and queue 47 small change batches while the worker is paused. Measure source age, view age, queue wait, bytes scanned and refresh duration. The lag budget should report a breach when the source is old even if the view build timestamp is recent. Preserve a dated last-good answer until a valid candidate catches up.

Gate the release

Compare the incremental candidate with the full rebuild by store-day key, row count and amount. Inject a failed index build: the table candidate may exist, but the reader pointer must remain on the old accepted generation. After repair, publish the table and index pointer together, query through the consumer route, and record the first generation that meets the age target.

Deliver an operating record

Submit source and view positions, refresh plan, changed-key inventory, correction and delete tests, full-versus-incremental comparison, backlog curve, cost per run, failed candidate, accepted pointer and consumer query. Include one rollback and one retry. A task-success message without a tested reader generation does not meet the project acceptance criteria.

Implementation

orders = {"order-47": ("west", "2026-10-01", 2375),
          "order-48": ("west", "2026-10-01", -125),
          "order-49": ("east", "2026-10-01", 6400)}

def store_day_totals(source):
    totals = {}
    for store, day, cents in source.values():
        key = (store, day)
        totals[key] = totals.get(key, 0) + cents
    return totals

assert store_day_totals(orders)[("west", "2026-10-01")] == 2250
orders.pop("order-48")
assert store_day_totals(orders)[("west", "2026-10-01")] == 2375

Performance and operating cost

The oracle rebuild uses O(N) time and O(G) state for N source records and G groups. Incremental refresh can narrow reads to changed keys, but requires an index from order identity to group and a safe path for deletes. The project intentionally retains a last-good snapshot and failed candidate, which increase temporary storage while making rollback and diagnosis possible.

Common Mistakes

Do not update only recent event dates when old refunds can change.
Do not expose a table candidate before the matching index is ready.
Do not call the freshness target met until the consumer query reads the new generation.