Cohort outcome monitoring: keep denominators and observation windows visible

Last updated: 6 Oct 20265 min read

tutorial

Group outcome rates are operational evidence only when the eligible population, label maturity and missing-group policy are explicit.

Define the harm and population

A delivery-claim triage model flags claims for manual review. The review team asks whether genuine claims from different service regions are being missed at different rates. False-negative rate among label-mature genuine claims addresses that question; overall approval or alert rate answers another. Define region from the approved request record, the eligible claim population, decision time and label-maturity interval before computing any gap. Selective labels can make reviewed cases look like the whole population when they are not.

Keep every denominator

For each region, store eligible cases, cases with mature ground truth, true positives, false negatives and unobserved outcomes. A rate without these counts hides how little evidence supports it. Treat missing region as an explicit unknown cohort with a count; do not assign it to the largest known region. If a cohort has no mature positive cases, its false-negative rate is undefined, not zero. Preserve protected or sensitive attributes only under approved access and retention. Logging limits still apply.

Compare like with like

Use the same time window, label definition, threshold and model version across groups. A rollout that routes regions to different revisions needs version-stratified analysis before a pooled comparison. Show uncertainty or at least minimum-count gates for small cohorts; one additional miss can move a tiny group’s rate sharply. An observed gap triggers review, not an automatic explanation of cause. Slice uncertainty and label correction records are part of the evidence trail.

Escalate a bounded finding

Tie the monitor to an owner who can examine label collection, threshold policy, regional operations and model error. Keep the current result as a dated measurement with its denominators and known blind spots. Do not declare a system fair from a single parity number. Release review compares candidate and incumbent on the same frame; the project catches a false improvement caused by missing outcomes.

Implementation

def false_negative_rate_by_region(cases):
    counts = {}
    for case in cases:
        region = case.get("region") or "unknown"
        bucket = counts.setdefault(region, {"positives": 0, "misses": 0,
                                            "unobserved": 0})
        if not case["label_mature"]:
            bucket["unobserved"] += 1
        elif case["genuine_claim"]:
            bucket["positives"] += 1
            bucket["misses"] += not case["flagged"]
    return {region: {**bucket, "false_negative_rate":
            bucket["misses"] / bucket["positives"] if bucket["positives"] else None}
            for region, bucket in counts.items()}

cases = [
    {"region": "north", "label_mature": True, "genuine_claim": True, "flagged": False},
    {"region": "north", "label_mature": True, "genuine_claim": True, "flagged": True},
    {"region": None, "label_mature": False, "genuine_claim": False, "flagged": False},
]
report = false_negative_rate_by_region(cases)
assert report["north"]["false_negative_rate"] == 0.5
assert report["unknown"]["false_negative_rate"] is None

Performance and operating cost

Scanning n cases costs O(n) time and O(g) storage for g region groups. At scale, preaggregated counts reduce dashboard work, but their provenance must retain model version and label window. Manual review of uncertain or missing outcomes costs more than the arithmetic and is necessary for a credible release decision.

Common Mistakes

Reporting group percentages without eligible and label-mature counts.
Treating a group with zero mature positive cases as a zero-error group.
Pooling model revisions and thresholds into one apparent trend.
Inferring the cause of a gap from the metric alone.