Schema registry and transitive compatibility

Last updated: 6 Oct 20265 min read

tutorial

A schema registry stores versioned event contracts; a compatibility rule decides which historic payloads a new reader can still decode.

Define the reader horizon

A payment topic can retain 47 days of events while consumers replay from their own older checkpoints. Checking a proposed schema only against the immediately previous version is insufficient if a reader can encounter any version in the retained log. Choose a transitive rule when the complete retained history must remain readable. Consumer rollout still needs separate semantic checks after wire compatibility passes.

Name the subject deliberately

A registry subject may follow a topic, a record type or another product boundary. The name controls which versions are compared and who may publish them. If unrelated events share one subject, a harmless change to one can block another. If related events use unrelated subjects, a consumer can miss a breaking shift. Record the subject strategy in the source contract.

Test actual old bytes

A rule saying a new field is optional does not prove that a production reader handles every older payload, nested default and unknown enum. Keep a small, authorized replay corpus with event bytes and schema identifiers from each supported version. Decode it in CI with the candidate reader. Measure whether the resulting business values still satisfy null, currency and event-time contracts.

Stage both deployment directions

Backward compatibility usually protects a new reader consuming old data; it does not automatically protect an old reader from a new writer. Roll out readers before producers when that is the safe direction for the chosen rule. A long-lived offline consumer may require a dual-publish window or a new topic. Owners and consumers should approve the semantic transition.

Stop irreversible changes early

Removing a required field, reusing a field identifier or changing cents to a floating currency value can break old data or corrupt meaning even when some decoders accept it. Register schemas in a controlled release step, reject incompatible versions before production writes, and preserve the schema ID with each event so replay never guesses which decoder to use.

Implementation

schema_versions = [
    {"version": 1, "required": {"payment_id", "amount_cents"}},
    {"version": 2, "required": {"payment_id", "amount_cents"}},
    {"version": 3, "required": {"payment_id", "amount_cents", "currency"}},
]

def reads_all_versions(reader_fields, historic_versions):
    return all(contract["required"].issubset(reader_fields)
               for contract in historic_versions)

assert reads_all_versions({"payment_id", "amount_cents", "currency"}, schema_versions)
assert not reads_all_versions({"payment_id", "currency"}, schema_versions)

Performance and operating cost

This simplified set check costs O(V × F) time for V retained versions and up to F required fields per version, with O(F) reader metadata. A real registry applies format-specific compatibility rules to full schemas, and a replay corpus also incurs decoding CPU and storage. Limit the corpus to representative supported versions while retaining enough history to catch a non-transitive break.

Common Mistakes

Do not assume a check against the latest version protects older retained data.
Do not equate wire compatibility with unchanged business meaning.
Do not deploy a new producer before checking old consumers under the selected rule.