Project: recover an observability pipeline without hiding user impact

Last updated: 5 Oct 20269 min read

project

Build a disposable receipt API, two scrape servers, a remote metrics store, an OpenTelemetry Collector tier, and a staging alert receiver. Send synthetic requests with known success, error, and latency outcomes. Record target discovery, local and remote request counts, export queue occupancy, sampled trace completeness, first-page latency, and the absence of test secrets before injecting faults. Keep test tokens non-privileged and the environment isolated from real on-call routes.

Break the measurement path

Make one exporter unreachable, then remove its job from discovery; prove that scrape-failure and absence checks respond differently. Issue slow and failed requests against two API replicas and calculate the fast-success fraction from a 470-millisecond histogram bucket. Stop the remote metrics receiver while local scraping continues. Record pending samples and WAL headroom, restore the receiver, and compare the same bounded counter range in local and remote queries. Degrade the trace destination separately: measure Collector queue growth and failed enqueues, restart a gateway with queued test trace IDs, and verify which IDs arrive after recovery.

Observability failure acceptance gates
Scrape: failed target and absent job detected separately
Latency: fast-success fraction matches known synthetic requests
Remote metrics: bounded interval reconciles after backlog drains
Collector: queue limit, retry expiry, and dropped spans recorded
Trace: retained slow and error traces contain expected spans
Alerts: root pages; only same-cluster symptoms inhibited
HA: request counter does not double with two scrapers
Privacy: canary token absent from exported signals and diagnostics

Test decisions and ownership

Split spans for one slow trace across sampling gateways, then restore trace-ID routing and compare trace completeness. Fire a database root alert with settlement symptoms in two clusters; only the matching cluster's symptoms may be inhibited. Take down one scraper and one Alertmanager peer and verify a page still arrives without doubling the remote request total. Inject a synthetic authorization token and customer email through the receipt path; inspect spans, logs, metric labels, exporter queues, and diagnostics before accepting the redaction boundary. Keep an independent synthetic request result as the user-impact reference whenever telemetry is missing.

Cost and verification

Estimate queue fill time from measured input and drain rates, not a guessed batch size. Record extra storage and network cost from duplicate scrapers, page delay from alert grouping, memory from trace decision waits, and the operational consequence of every missing interval. The drill passes only when the last trustworthy measurement interval is named, a page reaches its intended owner, and no canary token appears in exported data. A dashboard returning values is not enough.

Common Mistakes

Do not count absent targets as healthy requests.
Do not call a remote-write queue empty proof that every old sample arrived.
Do not route each span of one trace to a different stateful sampler.

Connected lessons

devops

project