Project: release review for visual part matching

Assemble the evaluation packet

A workshop wants technicians to find precedent images for a photographed seal defect. Freeze the defect rubric, component identity map, camera-period split, query set, gallery snapshot and encoder version. Keep views of each physical component on one side of the split. Document which examples were uncertain or adjudicated. The pair contract prevents label shortcuts.

Train against a credible baseline

Compare a fixed pretrained encoder with the trained triplet encoder using the same gallery and test identities. Inspect the chosen margin, mining refresh policy and reviewed false-negative rate. A model with lower triplet loss can still retrieve the wrong defect. Triplet loss and mining explain the training boundary.

Evaluate quality in layers

Measure exact-search task recall at 1 and 5, high-confidence wrong-repair rate, no-match rate and rare-defect slices. Then compare the approximate index to exact search on the frozen vectors, reporting neighbor recall and task recall after permission filters. Task recall and index recall cannot be collapsed into one score.

Test serving and rollback

Benchmark photo decode, encoder inference, index search, permission check and result rendering under peak load. Simulate a withdrawn reference and an encoder/gallery version mismatch. Keep a deterministic fallback that offers no unsafe repair instruction when confidence is low. Index lifecycle describes the swap and deletion path.

Record a reasoned decision

The sample gate below holds a candidate with a rare-defect regression even though its pooled task recall and latency pass. A later shadow period should validate logging and version coherence before a controlled rollout. Preserve previous encoder and gallery artifacts until rollback has been exercised.

Implementation

release_packet = {
    "task_recall_at_1": 0.84,
    "rare_defect_recall_at_1": 0.61,
    "exact_neighbor_recall_at_5": 0.97,
    "high_confidence_wrong_repair_rate": 0.008,
    "p95_full_path_ms": 139,
    "identity_disjoint_test": True,
    "permission_and_deletion_tested": True,
    "encoder_gallery_versions_match": True,
    "rollback_artifacts_kept": True,
}

def visual_match_release_gate(packet):
    holds = []
    if packet["task_recall_at_1"] < 0.82 or packet["rare_defect_recall_at_1"] < 0.68:
        holds.append("task recall by slice")
    if packet["exact_neighbor_recall_at_5"] < 0.95:
        holds.append("index agreement")
    if packet["high_confidence_wrong_repair_rate"] > 0.01 or packet["p95_full_path_ms"] > 145:
        holds.append("error or latency budget")
    if not all(packet[key] for key in (
        "identity_disjoint_test", "permission_and_deletion_tested",
        "encoder_gallery_versions_match", "rollback_artifacts_kept"
    )):
        holds.append("release controls")
    return "hold: " + ", ".join(holds) if holds else "eligible for shadow review"

assert visual_match_release_gate(release_packet) == "hold: task recall by slice"

Performance and operating cost

The gate is O(1); gathering the packet requires annotation review, encoder training, exact and approximate retrieval, full-path benchmarks and a deletion test. The example thresholds are local acceptance rules, not universal defaults. A shadow pass can reveal serving faults but does not prove field repair outcomes.

Common Mistakes

Do not promote from training loss or pooled recall alone.
Do not compare an approximate index with an exact oracle built from another encoder version.
Do not skip no-match, permission, deletion or rollback behavior.