Project: recover a saturated cloud boundary

Last updated: 5 Oct 20269 min read

project

Use a disposable account and cluster to run a claims recovery exercise. Create a synthetic receipt API, a queue worker, and a stub payment dependency with a controlled failure switch. Record normal transaction latency, provider connection count, gateway port errors, control-plane request rate, queue oldest age, and completed receipts before introducing faults. Keep all identifiers and payloads synthetic.

Introduce pressure at separate boundaries

Increase short-lived outbound connections to one destination while preserving a fixed business request rate. Confirm whether the shared egress path reports port-allocation errors; restore pooling and limit concurrent dials. Start several disposable infrastructure jobs and inject throttling in the provider stub. Show that each job respects one shared mutation budget and a total retry deadline. Calculate the maximum surge plus rollback demand for instances and subnet addresses before a canary. Refuse the rollout if the recovery reserve cannot be allocated.

Cloud boundary drill gates
Egress: real receipt succeeds; port errors return to zero
Rollout: request budget respected; no unbounded retry loop
Quota: surge plus rollback allocation proven
Queue: oldest eligible age falls with durable completions
Breaker: only live probes count as recovery
Bulkhead: renderer stall does not block settlement
Create: one intended resource and one state record after lost reply

Recover work without multiplying demand

Back up the queue with 700 synthetic receipts. Raise workers only to the measured database and provider rate ceiling, then verify that oldest eligible age decreases alongside durable completions. Induce a dependency timeout with client, proxy, and worker retries initially enabled; count total provider sends per operation. Assign a single retry owner and show that the same business key survives an ambiguous result. Open the dependency breaker, admit limited real probes, and keep fallback decisions labeled. Stall an optional renderer while proving the settlement bulkhead still serves requests. Finally drop a provider create response after acceptance; inventory and adopt the single created resource before resuming.

Cost and verification

Track extra API calls, compute time, queue deliveries, and temporary resource charges. Capture one timeline with original requests, retries, rejected admissions, completed receipts, and resource inventory. End the drill only when a synthetic customer receipt completes once, the queue age is within the chosen objective, the live dependency has passed probes, and the infrastructure record names exactly one intended resource. Remove disposable assets by their exact test identifiers after the evidence is retained.

Common Mistakes

Do not equate a smaller visible queue with completed work.
Do not call fallback traffic proof that the dependency recovered.
Do not repeat an uncertain create under a new identity.

Connected lessons

devops

project