Telemetry correlation and cardinality budgets

Last updated: 6 Oct 20265 min read

tutorial

A pipeline correlation key joins bounded logs, metrics and traces to the release being diagnosed without making every event identifier a metric label.

Give the release a stable context

Carry a logical interval, attempt ID and output generation through extraction, transformation, publication and serving. The identifiers let an operator jump from a freshness alert to the failed span and then to the run manifest. For an asynchronous queue, record the parent context or a link to the producing operation; clock-adjacent log lines are not enough to establish causality. Run lineage supplies data versions that request traces alone do not.

Keep metric labels bounded

A counter by pipeline and failure class has a useful, finite set of series. Adding order ID, file path or run ID as a metric label can create a new time series for every record and exhaust storage. Put high-cardinality identifiers in sampled traces or access-controlled logs, then retain aggregate metrics for alerting. Estimate the product of label cardinalities before adding a dimension.

Measure the consumer boundary

A green task metric only says a worker finished. Record the last accepted source interval, committed table generation, indexed serving generation and p95 consumer query latency. An index may lag a successful table release. The freshness contract must read from the generation the consumer sees, not from the latest successful extraction log.

Control sensitive context

Tracing baggage and log attributes can travel through services and exporters. Do not put customer names, raw payloads or secret tokens in correlation context. Use opaque IDs, restrict access to lookup tables and set retention by classification. Redaction after export does not undo exposure to an earlier collector. Test representative error paths because exceptions often log entire records.

Budget signal loss explicitly

A sampled trace is diagnostic evidence, not a complete audit record. Keep every committed release manifest and a small set of hard counters even if trace sampling drops routine spans. During an incident, temporarily raise tracing for a named pipeline without turning on global full capture. Validate that a failed publish emits a failure class, a bounded counter increment and no new serving generation.

Implementation

label_values = {"pipeline": 7, "stage": 6, "failure_class": 5}

def estimated_series(cardinalities):
    total = 1
    for count in cardinalities.values():
        total *= count
    return total

assert estimated_series(label_values) == 210
with_record_id = {**label_values, "record_id": 470000}
assert estimated_series(with_record_id) == 98700000

Performance and operating cost

Estimating series is O(L) time and O(1) auxiliary space for L label dimensions. Actual cardinality depends on observed combinations, but the upper-bound product catches expensive labels early. Trace storage grows with sampled operations and attribute bytes; metric storage grows with distinct series over retention windows. Keep audit manifests separately from sampled telemetry.

Common Mistakes

Do not place event IDs or file paths in metric labels.
Do not assume a trace sample is a complete publication audit.
Do not propagate sensitive payloads through trace context.