Project: release review for pump-risk explanations

Last updated: 5 Oct 20265 min read

project

An explanation release must state what the output describes, test its stability and support, and prevent model what-if scenarios from masquerading as repair instructions.

Freeze the prediction contract

A repair depot model routes pumps for manual review. Record the target event, prediction clock, feature and model versions, threshold, affected sites and operational owner. Keep a later-period holdout for model error and explanation checks. The explanation surface is visible only after a prediction exists; it cannot compensate for poor calibration or unsafe false negatives.

Compare multiple views

Use grouped held-out permutation importance for correlated sensors, ICE paths for approved temperature ranges, and a local model-fidelity check near the displayed case. Each answers a different question. Permutation, ICE and fidelity should not be collapsed into one “top reason.”

Audit actions separately

The user interface may show an approved inspection step, but a synthetic feature edit is not evidence that changing equipment state prevents failure. Test immutability, coupled sensor rules, action authority and no-solution behavior. Counterfactual feasibility defines those gates.

Inspect bad and unstable cases

Sample false negatives, rare pump families, new sensor revisions and predictions near the threshold. Ask whether the explanation remains consistent under harmless measurement changes and whether unsupported combinations were scored. Compare before and after retraining with version stamps. A pooled fidelity average may hide a dangerous slice.

Make a narrow promotion decision

The illustrative gate holds a release that passes local fidelity but contains an unsupported what-if scenario. A passed gate would authorize a controlled shadow review, not a guarantee that users will make better decisions. Keep the prior interface and model version available for rollback.

Implementation

explanation_packet = {
    "model_error_checked_by_site": True,
    "grouped_importance_reviewed": True,
    "ice_support_violations": 1,
    "local_fidelity_error": 0.008,
    "immutable_fields_locked": True,
    "no_solution_path_tested": True,
    "versioned_artifacts_retained": True,
}

def explanation_release_gate(packet):
    holds = []
    if not packet["model_error_checked_by_site"] or packet["local_fidelity_error"] > 0.02:
        holds.append("model error or fidelity")
    if not packet["grouped_importance_reviewed"] or packet["ice_support_violations"]:
        holds.append("unsupported explanation")
    if not all(packet[key] for key in (
        "immutable_fields_locked", "no_solution_path_tested", "versioned_artifacts_retained"
    )):
        holds.append("action or rollback controls")
    return "hold: " + ", ".join(holds) if holds else "eligible for shadow review"

assert explanation_release_gate(explanation_packet) == "hold: unsupported explanation"

Performance and operating cost

The gate is O(1). Producing its evidence requires held-out model scoring, repeated perturbations, support checks, human review of proposed actions and rollback drills. Explanation generation can add request latency and storage; measure those costs on the actual interface path before promotion.

Common Mistakes

Do not publish a model what-if as a verified repair action.
Do not approve attractive explanations while predictive error regresses.
Do not lose the model, feature, threshold and method versions needed to reproduce an explanation.