Kafka producer retries: preserve partition order without claiming end-to-end exactly-once

Last updated: 2 Oct 20267 min read

tutorial

Kafka producer idempotence uses producer identity and sequence numbers to prevent duplicate append from a retried send within its supported scope. It does not deduplicate a new application request, a new producer identity, or a payment charged outside Kafka. Ordering is scoped to one partition. With idempotence disabled, retrying an earlier failed batch while a later batch is in flight can reverse their arrival order. The producer configuration must therefore be reviewed alongside record keys, retry deadlines, transaction boundaries, and the consumer's external effect ledger. A producer success callback means the configured broker acknowledgment contract was met; it does not prove a database update happened.

Operational decision

An order service publishes status events keyed by order ID. Enable producer idempotence, compatible acknowledgments, and a finite delivery timeout. Record the business event ID in the payload. If a send outcome is ambiguous after a client timeout, retry the business operation through an outbox with a unique event ID rather than constructing a fresh, untracked event. A consumer that reserves inventory records the event ID in the same database transaction as its inventory change; a duplicate delivery then becomes a no-op. Transactions can atomically coordinate eligible Kafka writes and consumed offsets, but an HTTP call to a warehouse remains outside that Kafka transaction and needs its own idempotency contract.

Record key: order-7284
Business event ID: evt-38472
Producer idempotence: enabled
Kafka transaction boundary: Kafka records and offsets only
Inventory ledger unique key: evt-38472
HTTP warehouse call: separate idempotency key required

Cost and verification

Retries consume time, memory, network bandwidth, and broker capacity; high in-flight volume can enlarge the amount of work whose outcome is uncertain during a fault. A unique consumer ledger adds storage and indexed-write cost, but it makes replay and failover safer. Test a lost acknowledgment, a producer restart, and duplicate delivery separately. Measure both broker append duplicates and external effects. The two figures are different, and claiming exactly-once for the whole workflow when only Kafka writes are coordinated conceals a real billing or inventory risk.

Common Mistakes

Do not use a Kafka producer sequence number as a cross-service business ID.
Do not assume producer idempotence prevents duplicate HTTP or database side effects.
Do not infer global order across different partitions.

Connected lessons

Practice and check

devops

kafka

stream-operations