Production explanations: pin the exact decision and method

Last updated: 7 Oct 20265 min read

tutorial

An explanation is meaningful only for the model, features, policy and method that produced the decision under review.

Bind the explanation to the decision

A receipt reviewer may ask why a high-risk route was proposed. Store the decision ID, model digest, feature snapshot identity, preprocessing revision, score, policy revision, explanation method and method configuration. If the feature store changes before explanation runs, the resulting attribution may describe a different input. Restricted decision logs provide a stable join key; policy identity distinguishes a high score from the threshold that turned it into review.

Name what the method can show

A local feature attribution estimates how selected inputs relate to one prediction under a particular reference distribution or local approximation. It does not prove that changing a feature would cause the outcome to change. Correlated inputs can make an attribution unstable or misleading when a method perturbs them independently. Record reference data, approximation settings and validation diagnostics. If the method cannot support a claim, say so in the reviewer interface. The model card should list approved explanation uses and known limits.

Gate computation and display

Explanations may cost far more than scoring, so compute them asynchronously or for a bounded review subset when the latency budget requires it. Do not make a customer decision wait on an optional explanation that can time out. Validate that returned features belong to the scored feature contract, remove restricted attributes from broad display and refuse an explanation for a mismatched digest. Stability checks test whether the output is useful for the human task, not only whether it can be rendered.

Preserve reproducibility

Retain the explanation artifact or enough immutable inputs to rerun it under the same method version, subject to privacy retention. A later model promotion should not silently regenerate prior explanations under new bytes. Compare a sampled set of explanation outputs after deployment to detect changed method defaults or reference data. The audit project reconstructs a disputed receipt decision and rejects an attribution computed from the wrong feature snapshot.

Implementation

def explanation_identity(decision, explanation):
    required = ("model_digest", "feature_snapshot", "policy_revision")
    mismatched = [field for field in required
                  if decision.get(field) != explanation.get(field)]
    if mismatched:
        return {"state": "reject", "mismatched": mismatched}
    if not explanation.get("method_revision"):
        return {"state": "reject", "mismatched": ["method_revision"]}
    return {"state": "display", "mismatched": []}

decision = {"model_digest": "risk-47", "feature_snapshot": "snap-82",
            "policy_revision": "policy-3"}
explanation = {**decision, "method_revision": "local-r4"}
assert explanation_identity(decision, explanation)["state"] == "display"
assert explanation_identity(decision, {**explanation,
       "feature_snapshot": "snap-83"})["state"] == "reject"

Performance and operating cost

Identity comparison takes O(f) time and O(f) mismatch space for f manifest fields. The actual explanation cost depends on the method and number of model evaluations; an approximation may be much slower than inference. Retaining immutable feature evidence has storage and privacy cost, so use a defined retention window and access policy.

Common Mistakes

Explaining a historical decision with current feature values.
Treating feature attribution as a causal statement.
Displaying an explanation from a different model digest.
Allowing optional explanation work to consume the scoring deadline.