A flaky test produces different results for the same relevant source and test environment. It may expose a real race, shared state, timing assumption, or dependency outage. A successful rerun does not prove the first failure was harmless. Quarantine can protect delivery while the cause is fixed, but an unbounded exclusion quietly removes coverage from the release gate.
Make this comfortable
Flaky tests: quarantine one failure mode with an owner and expiry
Operational decision
A payout settlement test fails one run in twenty after parallel workers are added. Preserve the first failure's logs, seed, worker order, dependency versions, and revision. Reproduce with fixed code and varied order and concurrency, then classify whether the cause is application behavior, test isolation, or infrastructure. If the release gate must proceed, move only this test to a nonblocking quarantine lane with a named owner, issue, expiry date, and user risk statement. Keep it running on every relevant revision and alert when its failure rate rises; do not delete it or blanket-retry the entire suite until green. The contract below makes the exception reviewable. Fix the cause, require consecutive clean runs on the original gate configuration, and restore the test to blocking status. A new failure signature should reopen triage rather than inherit the old exception.
Settlement test quarantine record
Test: payout_does_not_commit_twice
Owner: ledger reliability team
Failure signature: parallel-run state collision
Release coverage: settlement smoke remains blocking
Quarantine: nonblocking run on every candidate
Expiry: next planned release window
Exit: isolate data; restore blocking gate after clean repetitionsCost and verification
Quarantine lowers immediate queue delay but carries a measurable coverage debt. Blind reruns increase runner cost and can mask an intermittent production defect. Track first-run failure rate, pass-on-rerun rate, age of each exception, and incidents linked to quarantined behavior. A short expiry forces review; a permanent exception merely renames a missing test. Keep evidence retention long enough to compare failure signatures without storing customer data.
Common Mistakes
- Do not treat a passing rerun as proof the original failure was noise.
- Do not quarantine an entire suite because one test is unstable.
- Do not leave a nonblocking test without owner, expiry, and replacement coverage.
Connected lessons
- DevOps: delivery, infrastructure, and reliable operations
- Continuous integration: test the merge candidate
- Incident reviews: turn a timeline into tested corrective work
- Policy exceptions: make a temporary bypass expire and prove its scope
- Distributed traces: preserve context without leaking data
Practice and check
devops
operations