Event schema evolution: release consumers before new event shapes

Last updated: 5 Oct 20266 min read

tutorial

An event schema is the wire contract shared by producers and consumers. A compatibility check can establish whether a reader can decode another schema version, but it cannot prove that business meaning stayed the same. Backward compatibility concerns a new consumer reading older events; forward compatibility concerns an older consumer reading newer events. Nontransitive checks may compare only with the immediately previous version, which is insufficient when consumers replay a long-lived topic.

Operational decision

A settlement stream adds an optional reconciliation code to payout events. Inventory consumers and the topic's retention window before producing that field. If old consumers must continue reading new events, select a compatibility rule that covers that direction and test old binaries against new messages. If a new consumer may replay events from the first retained revision, test against all retained schemas or use a transitive rule that fits the format. The text block is an approval contract, not a registry API request. Deploy consumers that tolerate the new field first, then the producer. Preserve an unknown-field behavior test and a business invariant: a missing reconciliation code must not silently mean the payout was approved. Run a replay sample before removing an old field; lagging consumers and dead-letter events may outlive the deployment window.

Payout event change gate
Field: reconciliation_code, optional on the wire
Old producer -> new consumer: verified against retained versions
New producer -> old consumer: verified for active readers
Business rule: absent value is not approval
Replay: oldest retained event decoded and processed
Remove old field only after consumer and retention inventory

Cost and verification

Registry checks add build time and version-management work but prevent many decode failures. A permissive rule can allow semantic breakage, while an overly strict rule can block safe additions. Retaining old consumers or dual fields uses engineering time and payload space. Track decode errors, consumer lag, and business-effect mismatches separately. A release that passes a schema check but sends the wrong amount is still a failed release.

Common Mistakes

Do not call a syntactic compatibility pass a business-contract proof.
Do not test only the newest prior version when replay spans older data.
Do not delete a field while a retained consumer still requires it.

Connected lessons

Practice and check

Advanced follow-up

CDC snapshot handoff: prove that initial rows and later changes form one history

devops

operations