Retries at multiple layers multiply request volume. If a caller makes three attempts and its proxy makes three attempts per call, a single operation can produce nine backend requests before a queue redelivery is counted. Each layer may look locally reasonable while the shared dependency receives a burst precisely when it has the least spare capacity.
Make this comfortable
Retry amplification: assign one owner for each failed operation
Operational decision
Trace one failed payout through the HTTP client, mesh proxy, provider SDK, and queue worker. Log an operation key, attempt number, and layer at each boundary; avoid logging the payment credential. Make one layer responsible for a bounded transient retry, and set a total deadline from the user's operation, not a fresh deadline for every attempt. Permit a retry only when the method and durable effect are safe under the original idempotency key. Return a final failure or park the work when the deadline expires. During a load test, inject a short provider outage and compare primary request rate, retry request rate, provider error rate, and completed payments. The contract below allows two total provider sends; a queue redelivery resumes the same business key and still checks whether the effect already committed. A retry budget should be reduced during overload rather than increasing attempts because failures are frequent.
Payout attempt contract
Business key: original payout operation ID
Retry owner: payout worker only
Provider sends: 2 maximum inside 12-second operation deadline
Proxy and SDK additional retries: disabled for this call
Transient response: bounded jitter before second send
Ambiguous result: read durable provider status before another sendCost and verification
A stricter retry limit may surface some transient errors sooner, but it protects recovery capacity and makes failure visible. Every extra attempt consumes network, CPU, provider quota, and sometimes money. Monitor retries as a fraction of original requests and count effects by business key, since a low error rate can hide duplicate writes. A no-retry decision is safe only when an explicit recovery path exists for incomplete work.
Common Mistakes
- Do not reset the deadline at each network hop.
- Do not count only application attempts when a proxy or SDK retries too.
- Do not retry an ambiguous side effect under a new operation key.
Connected lessons
- DevOps: delivery, infrastructure, and reliable operations
- Retries and timeouts: bound the cost of a failed request
- Queue consumers: acknowledgement, idempotency, and backlog
- Overload shedding: refuse excess work before latency collapses
- Queue visibility leases: prevent overlapping workers on one message
Practice and check
devops
operations