Abuse Decision Telemetry and False-Positive Rollback

Working case

The permit portal deploys a new export rule after a scraping incident. It blocks repeated downloads from one network address. A municipal office shares that address among 83 reviewers, and legitimate exports fall sharply. The security dashboard shows fewer downloads and initially calls the rule successful. Support tickets and task-completion data reveal the damage. The team narrows the rule to account and case behavior, restores a safe review route for affected users, and keeps the server authorization and per-tenant export budget intact.

Implementation boundary

function shouldRollBackRule(metrics) {
  return metrics.legitimateCompletionDrop > 0.08 || metrics.appealsPerHundred > 3;
}
console.log(shouldRollBackRule({ legitimateCompletionDrop: 0.11, appealsPerHundred: 2 }));
// Output: true

Record the decision stage, rule version, action type, coarse client category, and eventual task outcome. Use stable request correlation for short investigations, but do not expose tokens, document text, or raw private search terms in general logs. Provide a safe override or appeal process with expiry and audit. Stage new rules on observed traffic before enforcing, compare legitimate completion against abuse indicators, then increase enforcement gradually. Maintain a kill action for the new rule and a bounded fallback to older thresholds. Separate rule rollback from disabling all protection. Review the effect on shared networks, assistive technology, and low-traffic regions before declaring success.

Cost and boundaries

More telemetry adds write volume and analysis work. Aggregate by bounded categories instead of individual addresses or document IDs, and retain raw correlation only for a short investigation window. False positives have a direct support and trust cost, while too-loose rules increase compute and provider spend. A staged rule incurs temporary parallel evaluation, but this can prevent a broad outage. Measure denied requests, appeals, completed legitimate tasks, abuse cost, and time to disable a bad rule. Report both denominator and time window when comparing outcomes.

Failure trace

Enable the new export rule for a small cohort, then simulate 83 legitimate reviewers behind one address. The dashboard must show task completion falling, not only blocks rising. Roll back the rule version and prove ordinary export resumes while authorization and byte budgets stay active. Try an expired appeal token and require the standard decision. Send a burst from many addresses under one identity and confirm account-level controls still apply. Inspect logs for case content, recovery tokens, and unbounded high-cardinality labels. Force the telemetry sink to fail; request decisions should still follow a stated safe policy.

Verification

Rule outcomes include legitimate completion.
Rollback removes only the new rule layer.
Sensitive payloads stay out of routine telemetry.

Practice drill

Create an enforcement record with rule version 4, owner, cohort, stop threshold, and rollback action. Replay one day of bounded outcome counts from a shared office and a small remote region. Trigger a false-positive alert, disable only version 4, and verify the older rate and work budgets remain. Close the incident with the observed legitimate completions, abuse attempts, appeals, and retention deletion result.

Decision note

Defense success includes legitimate task completion and a narrow rollback path, not only the number of blocks.

Common Mistakes

Calling more blocks success without a denominator.
Using permanent allowlists for appeals.
Turning off all budgets to fix one bad rule.

Connected practice

Build Project: public permit endpoint abuse controls and review Web Development: jobs and abuse decisions quiz.