Disaster declaration: define authority, scope, and stop conditions

Last updated: 1 Oct 20266 min read

tutorial

The declaration contract names the impairment, affected user journey, recovery objective, evidence of standby readiness, expected data gap, and person authorized to move write authority. A partition can make two regions disagree about which side is healthy; an automated traffic switch without write fencing may create competing histories. A drill also needs stop conditions based on user impact, data integrity, and loss of rollback capability, with a separate procedure for halting fault injection and restoring protection.

Operational decision

A document-signing service loses requests from one network path while its primary database is still healthy. The incident lead checks independent client probes and origin telemetry before declaring a regional disaster. A local routing repair may be faster and less risky than promoting a replica. For an exercise, the team limits the target to a disposable tenant, chooses a maximum synthetic error rate, records who can stop the experiment, and verifies that the stop mechanism works from outside the impaired region. A stop alarm can end fault injection, but it does not automatically undo every side effect or restore a deleted resource; the runbook includes explicit cleanup and validation. If the write-fence result is unknown, freeze the promotion step even if the response-time target is threatened. If the standby is admissible and the primary impairment meets the declared threshold, the authorized operator records the chosen epoch and initiates the reviewed sequence. Another operator independently checks the target and intended direction. The team separates the decision to fail over from the automation that performs the well-tested steps, so an ambiguous signal can be examined without hand-editing infrastructure in panic.

Document-signing declaration
Impact: named customer journey and measured failure
Decision: repair locally or activate standby
Authority: incident lead plus independent checker
Hard stop: write fence unconfirmed or data gap exceeds bound
Exercise stop: synthetic error or integrity threshold crossed
Stop action: halt injection, inspect residual effects
Record: time, evidence, selected epoch, operator

Cost and verification

Decision gates add human minutes, but unbounded automatic promotion can add a much larger reconciliation cost. Record decision latency and false declarations alongside missed recovery objectives. For each exercise, account for probe volume, injected fault duration, cleanup time, and whether alarms and stop paths remained available during the simulated impairment. Practice the same authority path often enough that escalation is predictable.

Common Mistakes

Do not let one failing regional probe declare the entire region lost.
Do not assume stopping a fault experiment reverses its side effects.
Do not let the person choosing the target also be the only person verifying it.

Connected lessons

Practice and check

devops

disaster-recovery