Project: protect a release through failover and capacity pressure

Last updated: 5 Oct 20269 min read

project

This exercise protects a synthetic settlement service while its database, controller, image, and edge policy change. Use disposable infrastructure and test-only records. Capture the writer identity, last acknowledged payout, backup key, GitOps revision, image index digest, and baseline origin load before starting.

Stage and verify

Promote an isolated standby only after the former writer fails a controlled write attempt. Restore a retained recovery point with the intended recovery role, then decrypt a record written before key rotation. Apply a changed custom-resource definition and wait for establishment before introducing its dependent resource. Validate the controller processes a synthetic event. Build x86 and ARM image variants and run the same parser test on actual nodes of both types. Keep every digest and decision attached to the release record.

Settlement exercise evidence
Old writer rejected; new writer accepted one test payout
Old snapshot restored and historical record decrypted
Custom API established; controller processed test object
Both architecture variants ran the parser regression
Burst load: cache origin bounded and tenant budgets isolated
Recovery owner: named, with last acknowledged payout checked

Apply pressure

Expire a popular tariff key while forty-seven clients request it across several API replicas. Count origin reads and verify a legal-cutoff request never receives a stale tariff. Run two authenticated partners behind the same NAT address and confirm one cannot consume the other's budget. Apply a traffic profile with two bursts separated by a quiet period; observe autoscaler desired Pods, Ready endpoints, database sessions, and latency. If failover loses an acknowledged payout, stop the release and reconcile it before serving normal traffic.

Cost and verification

The drill uses extra replicas, protected backup storage, two image variants, and test traffic. Record the time until a serving endpoint appears, the number of origin reads per cache miss, rejected requests by tenant, and the oldest successful restore. Do not score the drill as passed solely because a controller reports healthy. Clean up disposable assets after retaining the evidence under the team's approved policy.

Common Mistakes

Do not declare failover safe while the former writer accepts direct clients.
Do not revoke the key needed to restore older copies.
Do not treat one platform's image test as proof for every node.

Connected lessons

devops

project