Project: case-save incident evidence

Last updated: 4 Oct 20268 min read

project

Build an evidence path for case 47 from browser submit through API acceptance, queue handoff, worker receipt, and final visible state. Propagate trace context through the queued message, but keep a separate durable operation ID for retries and business decisions. The incoming trace header is untrusted and never authorizes access. Define one eligible case-save operation and count it good only after the final state is visible within its deadline. Build a bounded metric label set and an allowlisted structured event; neither may include raw note text or reviewer email. Simulate release 29 accepting 62 writes while the receipt worker fails. A responder must see the real user impact, choose a reversible containment action, check compatibility before rollback, and verify both new saves and pending jobs after recovery.

Build contract

Join browser, API, and worker evidence with trace context plus a stable operation ID; prove collector loss does not change write acceptance.
Define the good, bad, and pending states for the case-save journey, deduplicate retries, and alert on sustained budget spend with a low-volume guard.
Inspect final telemetry exports for private content and bound metric labels to route template, release, and outcome category.
Record a first-impact and containment timeline; check queue-message and schema compatibility before rollback, then verify backlog repair.

Implementation checkpoint

function journeyOutcome(receipt, deadlineExceeded) {
  if (receipt === "visible") return "good";
  return deadlineExceeded ? "bad" : "pending";
}
console.log(journeyOutcome("queued", false));
// Output: pending

Cost and boundaries

For n case saves, a bounded event per transition uses O(n) record volume; tracing every internal span raises cost with span count and requires a sampling policy. Metric series should depend on a bounded number of route, release, and outcome combinations rather than grow with n case IDs. A queue backlog consumes storage while contained, but deleting it loses accepted work. A rollback may be fast only if old code can read messages and data written by the new release. Measure time to detect, time to contain, final operation completeness, and telemetry overhead under a synthetic burst. A quiet error chart after containment is insufficient when accepted jobs remain unresolved.

Failure drill

Send a forged trace header and prove it cannot select an account. Drop the collector, then verify the case-save contract still works while telemetry loss is visible. Return 202 for 62 writes and fail the worker: an HTTP-success indicator must not mark those journeys good. Add a case ID to a metric label and watch series count grow under 47,000 unique cases; remove it. Put a private note marker in an exception and check that no exported signal contains it. Roll back to a worker that cannot decode the new message and confirm the runbook blocks the action. Drain pending work and reconcile every accepted operation.

Acceptance checks

A complete operation can be traced without trusting a caller identifier for authorization.
Pending and failed terminal jobs reduce the user-journey indicator despite an API 202 response.
Telemetry export excludes private payloads and metric cardinality stays bounded.
Containment and rollback decisions include message compatibility and backlog verification.

Common Mistakes

Calling API acceptance a finished user task.
Keeping a private request body in a trace or exception event.
Rolling back code before checking queued-message compatibility.
Closing the incident while accepted jobs are still pending.

web-tech

web-development