Project: validate a case-volume forecast before it sets staffing

Last updated: 7 Oct 20265 min read

project

Archive true forecast origins, score multiweek errors and interval coverage, and hold staffing automation when outcomes or features leak across time.

Define the staffing decision

A service center sets weekly staffing from forecasts of incoming cases. Freeze which channels count, the intake timestamp, the roster decision date, each forecast horizon, and the loss of overstaffing versus an upper-tail shortfall. A forecast made Friday for a week several periods later cannot use the eventual schedule change, later marketing campaign, or finalized case count unless those facts were known Friday. The rolling-origin lesson enforces that chronology.

Archive and replay historical origins

Save the data snapshot, model version, target week, horizon, point prediction and interval generated at each origin. Evaluate candidates and a simple seasonal baseline on identical mature target weeks. Keep final origins untouched during selection. Reporting delays are real missing outcomes, not zeros. If the case taxonomy changed, map old and new definitions before claiming an improvement. The outcome ledger captures unresolved target periods.

Score the decision quantities

Report absolute error or another chosen point score, interval coverage and width by horizon, peak-week misses, and staffing shortfall cost. A model can reduce mean error while still missing the weeks that require surge staff. Check regions and queues separately with denominators. The interval lesson audits whether the uncertainty band is both calibrated and usable.

Gate rollout and monitor drift

The packet includes origin snapshots, feature availability times, target maturity, baseline comparison, horizon-specific scores, interval checks, and the cost threshold used to choose staffing. The gate below blocks automated recommendations when snapshots were reconstructed using future fields or interval coverage was never checked. Passing it starts monitored deployment; keep weekly residuals and route or channel mix under review as demand changes.

Implementation

def staffing_forecast_gate(audit):
    if audit["future_feature_leaks"]:
        return "hold:forecast-origin"
    if audit["pending_targets_as_zero"]:
        return "hold:target-maturity"
    if not audit["baseline_scored_on_same_weeks"]:
        return "hold:baseline-comparison"
    if not audit["horizon_coverage_reviewed"]:
        return "hold:interval-calibration"
    return "review:monitored-staffing"

audit = {"future_feature_leaks": 1, "pending_targets_as_zero": False,
         "baseline_scored_on_same_weeks": True,
         "horizon_coverage_reviewed": True}
assert staffing_forecast_gate(audit) == "hold:forecast-origin"
assert staffing_forecast_gate({**audit, "future_feature_leaks": 0})        == "review:monitored-staffing"

Performance and operating cost

The gate is O(1), while replaying k origins can require k model fits and O(k n) historical rows without caching. Timestamped snapshots consume storage, but without them a backtest can unknowingly use future data and overstate staffing value.

Common Mistakes

Recomputing historical features from a database after future corrections arrived.
Treating pending case totals as zero demand.
Comparing a model and baseline on different weeks.
Reporting mean accuracy while hiding costly peak-week staffing misses.