Project: release review for maintenance-search ranking

Freeze the decision packet

A municipal maintenance team wants to improve manual search. Record the candidate-index snapshot, query-log period, relevance rubric, adjudication sample, eligible document revisions and request split before tuning. The baseline is the current equipment-match rule. The candidate is a pairwise ranker. Keep the final later-period test unread until model and cutoff selection finish. The query contract is the input specification.

Build a two-layer scorecard

First measure whether retrieval brings at least one relevant eligible page into each judged request. Then, on identical candidate sets, compare NDCG at 3 and top-result grade by equipment slice. Report no-positive requests separately. A ranking gain with lower candidate recall is not a clean win. NDCG and candidate recall supply the calculations.

Audit the behavioral data

If clicks trained the candidate, inspect displayed-position logs, assignment probabilities, effective sample size and the proportion of unexposed documents. A manually judged holdout remains the primary offline relevance check. Do not convert historical clicks into a causal claim. The bias audit explains the limitation.

Test operational failure paths

Run permission tests against revoked documents, stale index revisions and a feature-service timeout. Benchmark p95 full-path latency under peak request concurrency, not only scorer time. Exercise the fallback heuristic and log fallback rates. An unsafe result at the top is a release blocker even if aggregate NDCG improves.

Make the release decision explicit

A shadow period can detect logging and latency faults without changing visible order. If it passes, a small randomized rollout can assess repair completion, time to approved instruction and safety guardrails with a predeclared stop rule. Rollout governance covers that step. The code below is a gate over a sample review packet, not a substitute for human inspection.

Implementation

review = {
    "candidate_recall": 0.91,
    "ndcg_at_3_delta": 0.034,
    "rare_equipment_ndcg_delta": -0.012,
    "p95_full_path_ms": 148,
    "permission_failures": 0,
    "fallback_exercised": True,
    "click_propensity_audited": True,
    "rollback_snapshot_retained": True,
}

def ranking_release_gate(packet):
    blockers = []
    if packet["candidate_recall"] < 0.90:
        blockers.append("candidate recall")
    if packet["ndcg_at_3_delta"] <= 0 or packet["rare_equipment_ndcg_delta"] < 0:
        blockers.append("ranking quality by slice")
    if packet["p95_full_path_ms"] > 150:
        blockers.append("request latency")
    if packet["permission_failures"] or not all(packet[key] for key in (
        "fallback_exercised", "click_propensity_audited", "rollback_snapshot_retained"
    )):
        blockers.append("operational controls")
    return "hold: " + ", ".join(blockers) if blockers else "eligible for controlled rollout"

assert ranking_release_gate(review) == "hold: ranking quality by slice"

Performance and operating cost

The gate itself is O(1); assembling its inputs requires human relevance judgments, query-level metrics, permission tests, realistic serving benchmarks and controlled traffic. A passing offline gate is permission to consider a measured rollout, not proof of improved field outcomes. Preserve the previous index and ranker artifacts until rollback has been exercised.

Common Mistakes

Do not promote on a pooled NDCG gain while rare equipment queries regress.
Do not use click-only labels as independent proof of relevance.
Do not start a rollout before permission, fallback and rollback paths are tested.