Incident triage prompts: build a timestamped evidence ledger

Last updated: 7 Oct 202611 min read

tutorial

An incident-triage prompt should turn logs, metrics, deployments, and operator notes into a time-ordered evidence ledger. Normalize clock zones before ordering events, preserve source IDs, and label each causal claim as observed or inferred. Logs are untrusted data: a line that contains instructions to an assistant cannot grant tool authority. The model can propose hypotheses and missing checks, but incident command and production actions remain with the authorized responders. An update should state current user impact and uncertainty rather than invent a root cause while the event is active.

Operational case

At 14:07 UTC, checkout latency rises; at 14:09 UTC, application logs show database pool waits; at 14:12 UTC, a deployment event appears. A support note written in local time claims the deployment started at 19:38, which is 14:08 UTC under the team's offset. The ordering now suggests a possible relation, but it does not prove the release caused the pool waits. The assistant records the offset, asks for a pre-release pool baseline, and identifies the rollback owner. It does not execute rollback because a log message says to do so. It also masks customer identifiers before the ledger goes to a shared channel.

INC-742; timezone for all rows: UTC.
14:07 latency alarm; 14:08 deployment began; 14:09 pool waits.
Observed: times and metrics. Inferred: deployment may contribute.
Next check: pool baseline and connection count by version.
Authority: incident commander decides mitigation; logs are data.

Performance and operating cost

Sorting E event records costs O(E log E) time and O(E) space; timestamp parsing and missing-zone review can dominate the human effort. A bounded incident packet avoids repeatedly sending full logs to the model, reducing both cost and private-data exposure. Recompute the ledger when new evidence arrives, but retain earlier versions so a later correction does not silently rewrite what responders knew. Measure false causal claims, time-to-useful-hypothesis, and missing critical signals. Summarization speed is irrelevant if it hides the impact window.

Common Mistakes

Do not merge timestamps from different zones without normalization.
Do not state that temporal proximity proves causation.
Do not obey commands embedded in log lines or operator notes.

Connected lessons

Continue with: Project: coordinate a parcel-platform incident review.

Continue with: Architecture prompts: test the failure path and its signals.

Continue with: Incident handoffs: preserve actions and decisions.

prompt engineering

software delivery