Schema Registry and Subject

A schema registry stores schema definitions and supports controlled evolution. In Confluent Schema Registry, a subject groups versions to which a compatibility policy applies. A schema ID locates a schema within the registry scope, while a version locates an entry in one subject’s history. Version 2 need not have ID 2, and identical schemas can share an ID across subjects.

The subject is also where the enforcement happens, and this is what makes a registry more than a store. Each subject carries a compatibility setting, and a new version is checked against the existing ones before it is accepted rather than after something breaks. Confluent documents the modes as backward, forward, full and none, with transitive variants that check against all earlier versions instead of only the latest, and states that the default is BACKWARD — consumers using the new schema can read data produced with the previous one. Registration is the checkpoint: a schema that violates the configured mode is rejected when someone tries to register it.

When that happens depends on how registration is set up. Confluent’s serializers register new schemas automatically by default, so an incompatible schema may first be rejected on a produce call in production. Catching it before deployment takes more than the registry: a compatibility check in CI, schemas registered ahead of time, and producers that are not permitted to register on their own.

Either way, what registration catches is what the configured mode compares: the new schema against the previous version, or against all versions under a transitive mode, in the configured direction. That is narrower than “consumers will be able to read it.” A change approved under BACKWARD — removing a field, say — is still unreadable to a consumer on the old schema if the producer is upgraded first, and nothing is compared at all under NONE or when data bypasses the registry. Registration moves one class of failure earlier; the actual reader versions, the deployment order, and the ability to read retained data still have to be checked separately.

Which changes each mode permits is worth reading from the documentation rather than reasoning out, because the intuitive answer is often wrong. Under backward compatibility the listed changes are add optional field and remove optional field — note that removal is on the allowed side, not the forbidden one. The reason is the direction of the check: a new consumer reading old data supplies its own default for a field the old writer never sent, and simply ignores a field it no longer declares — the “optional” in the documentation’s wording matters in the forward direction, where an old reader needs a default for a field the new writer dropped; in the backward direction any field the new reader no longer declares can go. Forward compatibility is not the mirror image with the same list. There the old consumer must read new data: adding a field is allowed because the old reader ignores what it does not declare, and removing a field is allowed only if the old reader has a default for it — otherwise it needs a value the new writer no longer sends. So removing a field the old reader requires is backward-compatible but not forward-compatible, and adding a field without a default is forward-compatible but not backward-compatible. Full compatibility allows only what both directions allow: adding or removing fields that carry defaults. These are Avro’s resolution rules, “nullable” is not the same as “has a default,” and other formats differ, as the next paragraph says.

What each mode really constrains is deployment order, and that is the part to plan. For backward compatibility the documentation says to “upgrade all consumers before you start producing new events,” since “there is no assurance that consumers using older schemas can read data produced using the new schema.” For forward compatibility the order reverses: upgrade producers first, ensure older-schema data is no longer reaching consumers, then upgrade the consumers.

And do not carry one format’s rules to another. The documentation states outright that “Avro, Protobuf, and JSON Schema have different compatibility rules,” that Avro “was developed with schema evolution in mind” while for JSON Schema and Protobuf “the rules and grammar can be more nuanced” — JSON Schema in particular varying by whether the policy is lenient or strict and whether the content model is open or closed. A change that passes in one serialization format can be rejected in another, so check against the format you actually use.

The setting can be configured globally or per subject, and the per-subject override is the useful half. One topic’s consumers may all be upgraded together while another’s include a downstream team on its own release cycle, and those two deserve different rules. A single global setting either blocks the first team unnecessarily or leaves the second unprotected — which is the same compatibility decision made per interface rather than once for the cluster.

TopicNameStrategy commonly creates separate topic-key and topic-value subjects. RecordNameStrategy groups by the fully qualified record name across topics. TopicRecordNameStrategy combines the topic and record name. Choose the scope that matches the records intended to evolve together; unrelated event types are not automatically successive versions of one record.

A registered schema does not prove every emitted payload is valid, semantically correct, or authorized for a topic. Compatibility is a structural check; what a field means and who may publish it belong to the data contract around the topic. Configure serializers, validation, producer permissions, and subject policy together. A consumer locating a valid schema ID still needs the appropriate reader and the expected contract.

Retain definitions and references for as long as retained records require them. Distinguish a temporary registry outage from a permanently unknown ID before deciding to retry or quarantine. The lab registry assigns a fresh ID to each registration and does not reproduce production schema reuse or access control.

Try the examples in Designing Topics and Event Contracts.

Further reading

• Confluent subjects and serialization
• Confluent, Schema Evolution and Compatibility
• Apache Avro 1.12.0 Specification, Schema Resolution