An idempotency key lets a server recognize a repeated mutating request after the client loses its response. The key must be scoped to the caller or operation, tied to a canonical payload identity, and stored atomically with the resulting business effect. A repeated key with different input is an error. A key record that expires too early can permit a late retry to create a second effect, while a cached failure may still need a precise policy for whether the client may start a new operation.
Make this comfortable
Idempotency keys: reconcile an accepted write before repeating it
Operational decision
A payout client sends a settlement request and loses the response after the ledger commits. It retries with the same account-scoped operation key and the same request fingerprint. The service must return the durable prior result or the current in-progress state without creating another payout. It must reject a retry that reuses that key for a different amount. Crash the service immediately after the ledger commit but before replying, then verify the key and payout are recoverable together; if they live in separate stores, a transaction or reconciliation protocol must close the gap. Test concurrent requests carrying the same key so one owns the effect and the other waits or receives a defined conflict. Record the key retention period against maximum client retry age, broker delay, and audit needs. Include an operation ID in the response and logs, but keep raw account details out of telemetry.
Payout idempotency contract
Scope: merchant account plus client operation key
Fingerprint: canonical amount, currency, destination, operation type
Atomic record: key, fingerprint, payout ID, state, durable response
Same key and fingerprint: return prior result or defined in-progress state
Same key and different fingerprint: reject without side effect
Expiry: later than every supported retry and replay windowCost and verification
Key records consume indexed storage and add contention for concurrent retries. A key held forever may outlive its useful audit period; a short retention window permits duplicate effects from late clients. The lookup is normally indexed, but lock waits and result serialization can dominate latency during a retry storm. Measure duplicate requests suppressed, mismatched-key errors, unresolved in-progress states, and payout count for each operation ID. Idempotency protects one defined operation; it does not make an entire multi-system workflow exactly once.
Common Mistakes
- Do not store the key after the payout has already committed in a separate uncoordinated step.
- Do not treat the same key with a changed amount as a harmless retry.
- Do not promise safe replay after the key has been pruned.
Connected lessons
- DevOps: delivery, infrastructure, and reliable operations
- Retry amplification: assign one owner for each failed operation
- Transactional outbox: commit business state and event intent together
- Ambiguous cloud creates: reconcile before repeating a timed-out mutation
- gRPC deadlines: spend one request budget across every downstream call
Practice and check
devops
operations