Schema-on-Read

Schema-on-read is an approach in which data is stored in the form it arrives, and a structure — column names, types, which fields matter — is applied only when someone reads it. Its counterpart, schema-on-write, requires data to fit a defined structure before it is accepted into storage.

Two places to put the decision

The distinction matters because a schema never really disappears. Someone still has to decide what a field means and what to do when it is missing or malformed. Schema-on-read moves that decision from the point of storage to the point of use.

Why it became popular

Traditional warehouses were schema-on-write: data was shaped for the business intelligence queries that would read it before it was loaded. That works when the questions are known in advance, and James Dixon’s 2010 description of the data lake named the limitation — a curated mart has examined only a subset of the attributes, so it can answer only the questions it was built for. The lake’s alternative was to keep data closer to its natural state and let different users explore it in different ways.

Cheap object storage and open file formats made that practical. Ingestion became simple, because nothing had to be modeled first, and exploratory work — data science in particular — could start from raw data rather than waiting for a warehouse team.

What it defers

The Lakehouse paper from CIDR 2021 describes the cost plainly: schema-on-read traded agility for deferring data quality and governance problems downstream. In practice that deferral shows up in three ways.

• Readers disagree. Two teams reading the same raw events can infer different types for the same field, or treat a missing value differently, and produce different numbers from identical data.
• Bad records fail late. A malformed record is not caught when it arrives but when a query touches it — possibly months later, in a report someone depends on.
• Inference is a guess. Tools that infer structure do so from what they see. Apache Spark, for example, can automatically infer the schema of a JSON dataset, and a samplingRatio option controls what fraction of records is used to do it; a rare field or type absent from the sample is simply not part of the inferred schema.

Even the choice of what to do with bad data becomes a reader’s setting. Spark’s JSON reader offers three modes: PERMISSIVE puts the malformed string into a designated corrupt-record column and sets unparseable fields to null — though with a user-supplied schema that lacks that column, it drops the record instead; DROPMALFORMED ignores malformed records entirely; and FAILFAST throws an error. The same file can therefore yield a count that keeps malformed rows, one that silently omits them, or a failure, depending on how the reader was configured.

Where each fits

Most platforms now use both, at different layers. Landing raw data schema-on-read preserves exactly what a source sent and makes reprocessing possible. Publishing curated tables schema-on-write gives consumers one agreed structure, enforced once. Lakehouse table formats point the same way: they keep data in open files but give each table a defined schema that writers are expected to respect, bringing back much of the protection of schema-on-write without giving up the lake’s storage model. How strictly a given engine enforces that on write is a setting to check rather than assume.

The useful question is therefore not which approach is better, but where in the flow structure should be enforced — and who is responsible for the data between arrival and that point. How warehouses, lakes, and lakehouses divide those responsibilities is worked through in Warehouse, Lake, and Lakehouse.