Serving indexes and freshness contracts

Last updated: 6 Oct 20265 min read

tutorial

A serving index organizes released data for specific queries; its freshness contract states which published generation each answer represents.

Start from a measured query

An operations dashboard needs account-day totals by account ID and recent day. A columnar fact table may scan efficiently for broad analytics but still be slow for one account. Build a keyed serving index or materialized view for that access pattern, and preserve the source snapshot ID with each indexed row. Fact grain determines whether one key maps to one result or many.

Publish index generations together

Loading an index record by record exposes mixed snapshots while the warehouse table stays atomic. Build a new index generation from one table snapshot, validate count and digest, then switch one reader pointer. During rebuild, the previous generation can continue serving with an explicit as-of time. Snapshot publication supplies the underlying transaction boundary.

Define acceptable staleness

Freshness is not simply job success. Measure source event time, table publish time, index publish time and query time. A dashboard may promise data no more than 18 minutes behind the latest accepted source interval. If that target fails, show the last-good timestamp or stop the query according to its use. Pipeline SLOs need the consumer-visible age.

Plan update and repair paths

Late corrections and dimension repairs can change prior account-day results. The index must replace the same logical key using a higher source generation, not append a second amount. Keep a reverse map or partition manifest to target affected keys. When a full rebuild is cheaper or safer, publish it atomically and compare a sample of queries against the warehouse snapshot.

Budget the extra copy

An index speeds selected queries at the cost of storage, refresh compute and another failure surface. Watch write amplification, hot keys, p95 query latency and index lag. If all consumers perform broad scans, a specialized index may add cost without benefit. The serving design should be justified by the actual read workload, not by a generic low-latency goal.

Implementation

warehouse_rows = [
    {"account": "acct-47", "day": 47, "cents": 5200},
    {"account": "acct-83", "day": 47, "cents": 1800},
]

def build_index(rows, snapshot_id):
    index = {}
    for row in rows:
        key = (row["account"], row["day"])
        if key in index:
            raise ValueError("account-day grain violated")
        index[key] = {"cents": row["cents"], "snapshot": snapshot_id}
    return index

serving = build_index(warehouse_rows, "snap-47")
assert serving[("acct-47", 47)] == {"cents": 5200, "snapshot": "snap-47"}

Performance and operating cost

Building a keyed index is O(N) expected time and O(N) extra storage for N warehouse rows; lookup is O(1) expected locally, subject to network and backend latency in production. Refresh work may duplicate the entire index per generation. Incremental updates reduce that cost but require stronger ordering and repair validation.

Common Mistakes

Do not serve a mixture of warehouse snapshot generations.
Do not claim freshness from a green job status alone.
Do not append a corrected key beside its old value.