Cohort baselines and denominator drift

Last updated: 6 Oct 20265 min read

tutorial

A cohort baseline compares a metric with a relevant historical population, while denominator checks distinguish true behavioral change from missing input.

Define the observation unit

A payment-success rate needs successful payment attempts divided by eligible attempts. Comparing raw success counts across days can mistake a source outage for a business decline. Name the denominator, exclusion rules, grain and event-time window before setting a threshold. Fact grain matters because retry events should not automatically count as new eligible attempts. Keep numerator and denominator in the published quality record.

Compare like with like

A Saturday overnight batch should not be judged against a weekday afternoon baseline. Segment by processing calendar, region, channel or another known demand driver, but require a minimum cohort size before trusting a narrow split. Too many tiny cohorts make the alert unstable and expensive. Record which reference windows were selected and which were excluded because of incidents or backfills.

Guard the input before the statistic

A percentage can remain plausible when both numerator and denominator lose the same shard. Check source completeness, unique-key count and expected partition coverage first. If a region delivered zero rows, mark the rate unavailable instead of reporting a reassuring zero-percent change. Reconciliation validates the input boundary separately from drift.

Use a bounded comparison

A median and median absolute deviation can resist one unusually large historic day, though a near-zero deviation needs a floor to avoid division by zero. Alert only when the absolute shift and relative shift matter to the consumer. Show actual value, baseline, threshold, cohort size and source generations together. A fixed global threshold can hide a serious change in a quiet region.

Retire stale baselines deliberately

A new payment channel or scheduled price change may shift the normal distribution. Do not silently train the incident into the baseline. Approve the new regime after source checks and business review, then version the reference window. Test the gate with a missing shard, a legitimate scheduled promotion and a real conversion drop; they should produce different outcomes.

Implementation

from statistics import median

historic_rates = [0.71, 0.73, 0.72, 0.74, 0.72, 0.73, 0.71]
eligible_attempts = 470
successful_attempts = 277

def rate_guard(successes, eligible, history):
    if eligible < 300:
        return "insufficient-input"
    observed = successes / eligible
    baseline = median(history)
    deviation = median(abs(rate - baseline) for rate in history)
    return "drift" if abs(observed - baseline) > max(0.07, 4 * deviation) else "pass"

assert rate_guard(successful_attempts, eligible_attempts, historic_rates) == "drift"
assert rate_guard(0, 0, historic_rates) == "insufficient-input"

Performance and operating cost

Sorting for a median costs O(H log H) time and O(H) space for H historical rates in this reference implementation. A production monitor can maintain compact summaries, but should preserve enough raw counts to audit a disputed alert. Every additional cohort multiplies retained state and test volume; choose segments that correspond to real consumer or source boundaries.

Common Mistakes

Do not compare raw counts when the eligible population changed.
Do not report a normal rate from incomplete source partitions.
Do not auto-refresh a baseline with an unresolved incident.