A proxy can send a copy of a production request to a shadow backend while returning the primary backend's response to the client. The shadow response is not a rollback mechanism, and the mirrored request can still mutate databases, publish events, spend money, or send messages. A safe shadow environment needs separate credentials and destinations, write blocking or disposable data, and an explicit privacy boundary for copied payloads.
Make this comfortable
Shadow traffic: test a new backend without duplicating live side effects
Operational decision
A receipt read API is being replaced. Mirror a bounded fraction of read requests to an isolated candidate with a sanitized data snapshot. The route fragment shows the intent; it is not a complete proxy configuration. Confirm the mirror cluster exists and receives traffic, then compare status, selected response fields, latency, and errors by request ID without returning candidate output to users. Do not mirror settlement POST requests until their entire effect graph is replaced by test sinks or rejected at the candidate edge. Test a request containing a synthetic customer marker and verify it appears only in approved shadow storage. Limit shadow rate independently from primary traffic so a slow candidate does not exhaust shared egress or backend capacity. A primary request can succeed even if the shadow request fails; monitor the candidate separately. Stop the mirror if isolation, redaction, or cost gates fail.
route:
cluster: receipt-primary
request_mirror_policies:
- cluster: receipt-shadowCost and verification
Mirroring spends extra network, CPU, storage, and possibly third-party quota while providing no direct user response benefit. Sampling a small fraction reduces cost but may miss rare request shapes. A slow shadow path should not delay primary responses, but it can still compete for shared infrastructure and logging capacity. Measure primary latency before and after enabling the mirror, candidate comparison coverage, and every unexpected shadow write. Treat a successful comparison as rollout evidence, not proof that all mutating routes are safe.
Common Mistakes
- Do not mirror production writes into a candidate connected to real payment or email systems.
- Do not assume the client can see or validate the shadow response.
- Do not let untrusted request headers choose an arbitrary shadow destination.
Connected lessons
- DevOps: delivery, infrastructure, and reliable operations
- Progressive delivery: canary checks and rollback
- Canary analysis: compare a small cohort without hiding its failures
- Telemetry redaction: remove sensitive fields before an exporter or sampler sees them
- Idempotency keys: reconcile an accepted write before repeating it
Practice and check
devops
operations