Model incidents: build a release and evidence timeline before rollback

Last updated: 6 Oct 20265 min read

tutorial

An incident timeline ties customer symptoms to model, feature, data and infrastructure changes so mitigation targets the failing boundary.

Start with the visible failure

A receipt service may show higher manual-review volume after a model release. First determine whether the cause is model decisions, feature admission, source lag, queue saturation or a downstream review outage. Record when the symptom began and its customer scope. Compare model digest, feature-contract version, producer revision and serving image around that time. Actionable alerts provide the entry point, not the diagnosis.

Keep event time and observation time separate

A data pipeline can publish late records whose event timestamps are old. A chart grouped only by event time may make a failure appear to precede the deployment that caused it. Retain deployment time, data availability time, request time and alert observation time. Annotate metric windows and sample counts. When labels are delayed, do not claim quality recovered because the most recent window has few outcomes. Mature evaluation determines when quality evidence is usable.

Mitigate by reversing the responsible change

If the candidate model caused errors, restore the known-good model and compatible feature producer. If the feature store is stale, rolling back the model alone may leave the outage unchanged. If capacity is exhausted, limit intake and use the approved fallback while scaling or repairing the dependency. Record the exact action, actor, timestamp and pre/post metrics. Promotion evidence tells the responder which old digest is loadable; fallback policy defines a safe customer outcome.

Close with an evidence-backed review

After stabilization, compare request outcomes, error counts and mature quality cohorts. Distinguish a mitigation that reduced errors from a root-cause repair. Keep the incident timeline and the smallest set of restricted examples needed for review under privacy policy. The receipt drill tests a false model alarm caused by a feature producer, so responders must avoid a reflexive but ineffective model rollback.

Implementation

def incident_changes(events, symptom_at):
    allowed = {"model", "feature-producer", "serving-image", "data-source"}
    return sorted((event for event in events
                   if event["kind"] in allowed and
                   event["deployed_at"] <= symptom_at),
                  key=lambda event: event["deployed_at"], reverse=True)

events = [{"kind": "model", "deployed_at": 17, "revision": "r8"},
          {"kind": "feature-producer", "deployed_at": 21, "revision": "f4"},
          {"kind": "model", "deployed_at": 29, "revision": "r9"}]
assert [event["revision"] for event in incident_changes(events, 24)] == [
    "f4", "r8"]

Performance and operating cost

Filtering e release events takes O(e) time; sorting matches takes O(k log k) time and O(k) space. A production timeline also needs clock normalization and provenance checks. Temporal proximity is a clue, not proof of causation; compare affected and unaffected traffic and test the suspected boundary before declaring a root cause.

Common Mistakes

Rolling back the model when a feature producer is broken.
Treating old event timestamps as deployment chronology.
Claiming quality recovered from immature labels.
Closing an incident after mitigation without recording the root-cause repair.