Probe alert quality: coverage, noise and failure rehearsal

Last updated: 6 Oct 20265 min read

tutorial

A probe suite needs explicit route coverage, alert thresholds and rehearsed failure cases before it can be trusted on call.

Map every probe to a failure boundary

List the service paths that matter: parse request, fetch online features, load model, apply policy and produce a response before deadline. Map each synthetic request to the steps it traverses and the failure it should catch. An all-green dashboard can still omit the fallback route or one region. Keep a coverage ledger with owner, last successful run and current model revision. Trace coverage gives sampled production evidence; a probe ledger states which paths are deliberately exercised.

Avoid alerting on one weak signal

A single timeout may be a transient network interruption; a sequence of route mismatches may be a real release fault. Set alert policy by route and expected run frequency, with a bounded confirmation window and a severe path for an unsafe release decision. Do not average a failed quarantine probe with healthy ordinary probes. Record the exact fixture, region, dependency revision and response. Metric identities keep revisions separate rather than hiding an issue behind one pooled series.

Rehearse known failures

Before relying on a new suite, deliberately point a canary at a stale feature snapshot, a mismatched model revision and a slow dependency. Confirm that the expected probe fails, an alert reaches the correct owner and the runbook names a reversible action. Run the rehearsal in a controlled environment or dedicated canary cohort, not by corrupting live customer traffic. Incident evidence preserves the timeline for the real event.

Measure alert utility

Track true incidents caught, false alerts, time to detection, unattended failures and the age of probe fixtures. Retire fixtures that no longer represent a valid request, and add new ones when the response contract or feature source changes. Suppress a known fixture failure with an expiry and owner rather than muting the entire suite. The drill catches a probe that stayed green because it never read the live feature store.

Implementation

def probe_alert(window, policy):
    if window["unsafe_route_mismatches"]:
        return "page:unsafe-route"
    if window["scheduled_runs"] < policy["minimum_runs"]:
        return "investigate:missing-runs"
    if window["consecutive_failures"] >= policy["failure_streak"]:
        return "page:repeated-failure"
    return "observe"

policy = {"minimum_runs": 3, "failure_streak": 2}
window = {"unsafe_route_mismatches": 0, "scheduled_runs": 3,
          "consecutive_failures": 2}
assert probe_alert(window, policy) == "page:repeated-failure"
assert probe_alert({**window, "unsafe_route_mismatches": 1}, policy)        == "page:unsafe-route"

Performance and operating cost

Alert evaluation is O(1) time and space after window aggregation. Probe execution cost scales with run frequency and regions; confirmation windows reduce noise but lengthen detection. A safety-critical route should use a tighter alert policy than an ordinary latency variation, with the extra page load reviewed regularly.

Common Mistakes

Averaging a failed critical route with healthy ordinary routes.
Counting missing scheduled runs as healthy because no failures arrived.
Muting an entire suite indefinitely for one obsolete fixture.
Assuming a probe exercises a dependency without verifying its route trace.