Synthetic inference probes: test the full route without polluting outcomes

Last updated: 6 Oct 20265 min read

tutorial

A production probe is a controlled request with a pinned expected envelope, isolated from customer decisions and training feedback.

Define a probe as a contract

A cold-chain spoilage model routes sensor readings to quarantine or release. A ping to the HTTP process proves only that the process answered; it does not exercise parsing, feature retrieval, model loading, policy routing or response formatting. Create a small approved set of synthetic sensor records with expected schema, model tuple, route envelope and maximum latency. A probe may tolerate a score interval rather than demand an identical float across hardware. Response contracts identify the fields that must survive the route.

Isolate probe effects

Mark each request with a non-user probe identity before it enters the serving path. Route its result to a separate telemetry stream and suppress any side effect that would release a real shipment, open a customer case, send a notification or join a training label. Keep a narrowly scoped allowlist of probe IDs and prevent clients from asserting that marker themselves. A synthetic request that enters outcome monitoring can manufacture apparent model success. Outcome joins should reject probe identities explicitly.

Pick probes for failure boundaries

Use one ordinary sensor record, one missing-feature case and one boundary case near the quarantine threshold. Add an expired feature snapshot to verify fallback behavior. Version the probe inputs alongside the model and feature contract: a valid expected result can change after an approved threshold release. Run probes from more than one region if the service can fail over. Failover identity guards against a standby running an old model or feature revision.

Keep probes distinct from quality evidence

Passing probes confirms that chosen paths still behave within their contract. It does not establish population accuracy or catch an unseen cohort. Track probe failures, route mismatches and latency, then compare with representative production telemetry and mature outcomes. A bad probe may also fail because its fixture expired; treat that as a fixture issue until diagnosed. Probe alert design makes these limits visible, and the project tests a stale-feature fallback.

Implementation

def probe_result(request, response, contract):
    if request["identity"] != contract["probe_identity"]:
        return "reject:identity"
    if response["model_revision"] != contract["model_revision"]:
        return "fail:model-revision"
    if response["route"] not in contract["allowed_routes"]:
        return "fail:route"
    if response["latency_ms"] > contract["deadline_ms"]:
        return "fail:latency"
    return "pass"

contract = {"probe_identity": "cold-chain-probe-r47",
            "model_revision": "spoilage-r8", "allowed_routes": {"quarantine"},
            "deadline_ms": 82}
request = {"identity": "cold-chain-probe-r47"}
response = {"model_revision": "spoilage-r8", "route": "quarantine",
            "latency_ms": 46}
assert probe_result(request, response, contract) == "pass"
assert probe_result(request, {**response, "route": "release"}, contract)        == "fail:route"

Performance and operating cost

Contract evaluation is O(1) expected time and space for a fixed route set. Each scheduled probe consumes an inference and possibly a feature read. A large probe fleet can distort capacity and cost measurements, so rate-limit it, label it in telemetry and schedule enough runs to detect outages within the required response window.

Common Mistakes

Calling an HTTP 200 response proof that the model path is healthy.
Letting synthetic decisions enter customer actions or training labels.
Allowing arbitrary clients to set the trusted probe marker.
Treating a handful of passing fixtures as evidence of model accuracy.