Warehouse, Lake, and Lakehouse

A retailer’s data team is deciding where a new dataset should live. The clickstream is already landing in object storage as files. Finance wants it joined to orders in the warehouse, where the revenue tables are. The machine learning team wants to read it directly, without going through a SQL connector. Somebody points out that the company is paying to store the order data twice. Somebody else points out that the last time a table was written by two jobs at once, the report was wrong for a day and nobody noticed. The retailer is invented, but every part of that argument is a real one.

Underneath it are three architectures that were each designed to fix what the previous one got wrong, and each of which deferred a different problem to somebody else. This article works through what the warehouse, the lake, and the lakehouse were built to solve, what assumption each one made, why the two-tier combination became the default despite its cost, why transactions turned out to be the hard part, and why the label on a product now tells you much less about its behavior than it used to.

Three answers, three assumptions

The order matters, because each design is a response to the one before it.

The data warehouse came first, to give business leaders analytical answers by collecting data out of operational databases into one place for decision support and business intelligence. The Lakehouse paper from CIDR 2021 characterizes these first-generation platforms by two properties. Data was written schema-on-write, so the model was shaped for the BI queries that would read it. And compute and storage were coupled in an on-premises appliance, which meant an enterprise had to provision and pay for the peak of both user load and data volume.

These designs served established reporting workloads, but changing inputs and workloads could make the original model or appliance capacity a poor fit. A predefined schema does not prevent new questions: retained detail can support new filters, groupings, and joins. Questions become impossible when the necessary attributes or historical detail were never kept.

The data lake was the answer to that. James Dixon, who coined the term on his blog in October 2010, framed it as a contrast with the data mart, which he described as a store of “bottled water — cleansed and packaged and structured for easy consumption.” His objection concerned the information lost in packaging: keeping only selected attributes and aggregates limits which later questions the mart can answer. His lake holds the contents in a more natural state, streaming in from the source, with different communities of users coming to examine it, dive in, or take samples.

Translated into architecture, the lake meant low-cost storage with a file API holding data in generic, usually open formats such as Parquet and ORC. It started on HDFS and, from about 2015, moved to cloud object stores, which the Lakehouse paper credits with better durability, geo-replication, and low cost. The trade it made was explicit in the name schema-on-read: applying a table schema when reading the stored data. That can defer modeling, but it does not require deferring all quality checks or access controls. The unmanaged case described by the paper leaves readers to resolve those problems; a lake can instead validate arrivals, quarantine bad records, and provide shared schemas and controlled access.

Neither structure replaced the other. What actually happened is that organizations built both and connected them. Writing in 2021, the Lakehouse authors reported from their own experience that this two-tier arrangement was used at virtually all Fortune 500 enterprises; how common it remains is not something that paper can settle, and it is worth treating as a description of that moment rather than a current statistic.

Structure	Problem it addressed	Assumption it made	What it deferred
Warehouse	Scattered operational data could not answer business questions	Model and validate data before loading for the intended workloads	Questions requiring attributes or detail omitted from the stored model
Lake	Storing everything was too expensive and too rigid	Store first in open formats, interpret later	Interpretation at read time; validation, governance, and transactional publication still need explicit mechanisms
Two-tier (lake feeding warehouse)	Neither structure served every user	Copying the curated subset into a warehouse is an acceptable cost	Consistency between the copies, and the freshness lost in the second hop
Lakehouse	The copy and the gap between the two tiers	A metadata layer over open files can provide warehouse-grade management	The operational work of running that layer, which is real and ongoing

What the two-tier arrangement costs

In the two-tier pattern, data is extracted into the lake and then loaded again into the warehouse for the reporting that matters most. It looks economical, because cheap object storage holds the bulk and the expensive engine holds only the curated part. The Lakehouse paper lists four costs of that arrangement, and they are worth separating because they land on different people.

• Reliability. Keeping the two consistent takes continuous engineering. Every extra step is a place where a run can fail or a subtle difference between the two engines can change a value, which is one reason reconciliation between the tiers is not optional.
• Staleness. The warehouse copy lags the lake, sometimes by days. The paper points out that this is a regression from first-generation systems: there, data was loaded once from the operational systems into the warehouse and was queryable from that point, with no second hop from a lake to a warehouse. The first load still took whatever it took; what the two-tier design added was another stage behind it. It cites a Dimensional Research and Fivetran survey reporting that 86 percent of analysts work with out-of-date data and 62 percent wait on engineering resources several times a month. Those figures come from a survey the paper cites rather than from its own measurement.
• Poor fit for advanced analytics. As of the paper, the leading machine learning systems did not work well on top of warehouses: they need large volumes through direct file access and non-SQL code, and reading that through ODBC or JDBC was inefficient with no way to reach the warehouse’s internal format. The usual workaround was to export from the warehouse back to files, adding a third hop and more staleness; running those systems against lake files instead meant giving up transactions, versioning, and indexing. Warehouse vendors have since added ways to read data more directly, so the sharpness of this constraint depends on the product and version in front of you.
• Total cost of ownership. The pipeline between the tiers has to be paid for continuously, the duplicated data is paid for twice, and data held in a proprietary format raises the cost of ever moving it.

These costs overlap. Reconciliation, retries, exports, and extra copies can all consume billed compute, storage, or transfer, as well as engineering time. A cost review should separate resource charges from staff effort and the consequences of stale or inconsistent results, then avoid counting the same work twice. Which item dominates depends on data volume, workload, service pricing, and operating practice; duplicated storage has no fixed place in that ranking.

Transactions were the hard part

The obvious response to the two-tier cost is to serve both audiences from one copy in open formats. That idea is older than the word lakehouse, and it kept failing for a specific reason: object stores do not give you a table.

As the paper puts it, storage systems such as S3 or HDFS provide a low-level object store or file system interface, in which even a simple operation like updating a table that spans multiple files is not atomic. A job that rewrites forty files and fails after twenty leaves a directory that is neither the old table nor the new one. A reader that lists the directory while a writer is halfway through sees a mixture. There is no commit, so there is nothing to roll back to.

The fix was to add a layer that records which files constitute a table at a given version, and to make updating that record the atomic operation. The paper traces the lineage.

System	Origin	Where the table state lives	Noted characteristic
Apache Hive ACID	The starting point for this approach	An OLTP database outside the lake	Transactional updates to the file set, with the external database as a scaling limit
Delta Lake	Databricks, development begun 2016	A transaction log in the lake itself: a `_delta_log` directory of newline-delimited JSON commit files, numbered by version, periodically summarized into Parquet checkpoints	Scales to billions of objects per table; an existing Parquet directory can be converted with no copying
Apache Iceberg	Started at Netflix	Table metadata in the lake, organized as metadata files, snapshots, and manifests	Supports both Parquet and ORC storage
Apache Hudi	Started at Uber	Similar design, inside the lake	Focused on simplifying streaming ingest; did not support concurrent writers as of the paper

Versioned publication lets a writer expose its changes atomically and lets readers use a consistent table snapshot. Keeping older versions addressable supports time travel: querying a retained past state. Zero-copy cloning adds a different operation, creating a separate object that initially shares stored data. Snowflake, for example, allows the original and clone to change independently and gives each its own lifecycle. Snapshot retention alone does not supply that operation; clone support, permissions, and the effect of source cleanup depend on the implementation.

The shared principle is that a writer stages its data files and then publishes a new table version atomically, so a reader sees either the previous version or the new one. That is atomic publication combined with snapshot reads of the MVCC kind. How the publication is made atomic differs by format, and the difference matters when operating them. Delta commits by writing the next numbered JSON file in the log directory, so atomicity comes from only one writer being able to claim a given version number; readers replay the log from the most recent checkpoint. Iceberg maintains table metadata, snapshots, and manifest files and commits by making the current metadata authoritative through its own mechanism. Both reach the same guarantee, and neither behaves quite like the other under contention or maintenance, so the operating team should read the specification of the format it actually runs.

Correctness was necessary but not sufficient. A table that is transactional and slow does not replace a warehouse. The paper’s performance argument is that a system can keep the open base format and still go fast by maintaining things it does control: column min and max statistics per file, which let the engine skip files that cannot contain a match; Bloom-filter indexes; record ordering, including space-filling curves such as Z-order for locality across several dimensions; and caching of the hot subset. These are the same ideas behind partition pruning and predicate pushdown, applied at the level of the table’s metadata rather than the file format.

Three operational consequences follow from this design and are easy to miss when reading the architecture diagram.

• The metadata layer is something to run. Snapshots accumulate and need expiring, and the many small files that frequent writes produce need compaction or queries slow down. Both are scheduled jobs that somebody owns.
• Concurrency is optimistic. Writers that rarely collide barely notice; many jobs writing the same table at once will make each other retry.
• Deleting a row does not delete the data. A delete produces a new table version in which the row is absent, but earlier versions still reference the file that contains it. Some formats go further and do not rewrite the file even for the current version: Delta’s deletion vectors mark rows as removed while leaving the Parquet file in place, so the deleted bytes are still present in the file the current version reads. Either way, removal of the bytes is a separate, later operation.

That last point has a compliance consequence worth stating early, because it surprises teams late. A request to erase personal data is not satisfied by a DELETE statement on a table format, and the steps that follow depend on the format.

• Where deletions were recorded as vectors, the data file must first be rewritten without the deleted rows. Delta’s documentation describes REORG TABLE ... APPLY (PURGE) as rewriting the data files that contain deletion-vector modifications, and notes that this creates a new table version whose timestamp is the one to consider for the retention threshold that follows.
• Then the superseded files must be physically removed, which is a separate command with a retention period — VACUUM in Delta’s case — and the files remain readable by older versions until it runs and that period has passed.
• Where old versions are kept as snapshots, those snapshots also have to be expired before the files they reference become unreferenced and eligible for removal.

Stated generally, erasure means making sure no file still holds the bytes and no retained version still references one that does: rewrite the affected files where the delete did not already do so, then remove the superseded files once their retention and reference conditions are met. Which of those steps your delete already performed depends on the format and its settings, so the procedure has to be written against the format actually in use and tested rather than assumed. Time travel and erasure pull against each other, and the retention setting is where the two are reconciled: a retention window longer than the erasure deadline is a compliance problem no amount of tooling fixes.

The labels stopped describing the implementations

The strongest practical reason not to choose by name is that the categories converged while the names stayed put.

Consider a product usually filed under “warehouse.” Snowflake’s own documentation, checked in September 2026, describes an architecture in three layers: database storage, query processing through virtual warehouses, and a cloud services layer for authentication, metadata, and query optimization. It calls the design a hybrid of shared-disk and shared-nothing. Storage is reorganized into an internally optimized, compressed, columnar form divided into micro-partitions. Compute clusters are independent of one another. And the table types it lists include Apache Iceberg tables alongside its own.

Almost nothing in that description matches the first-generation warehouse of coupled compute and storage. To be fair to the paper, it does not claim otherwise: it discusses cloud warehouses that already separate storage and compute, and argues against putting everything in one on the grounds that such a design still does not handle video, audio, and text well or give machine learning fast direct access. What has changed since is that several of those specific gaps have narrowed, as warehouses added open table formats and more direct read paths. The useful question is therefore not whether the paper was right in 2021 but which of the constraints it listed still bind for the product and version in front of you.

The convergence runs both ways. A “lakehouse” that nobody compacts, whose snapshots are never expired, and whose catalog is a wiki page behaves like a lake with extra ceremony.

The useful move is to stop asking what a system is called and ask what it does. Seven properties separate real systems more reliably than the three nouns do.

Property	Question to ask	Why it matters
Format ownership	Can another engine read the base data without going through this product, and does it also understand the table layer — current snapshot, delete files, deletion vectors?	Determines migration cost and whether other tools read correctly. Reading the directory’s Parquet files without interpreting the table metadata returns rows from older versions and rows that were deleted
Transaction unit	What exactly is atomic: a file, a partition, a table, several tables?	Determines what a half-finished job leaves behind
Isolation for readers	What does a query see while a load is running?	Determines whether reports can run during ingestion
Concurrent writers	What happens when two jobs write the same table?	Optimistic schemes degrade sharply under collision
Compute and storage coupling	Can they be scaled and paid for separately?	Determines whether peak load sets the permanent cost
Workload isolation	Can a heavy job slow down an unrelated one?	Determines whether BI and data science can share a platform
Metadata and governance	Where do schema, lineage, and access policy live, and are they enforced for every engine?	A policy that only one engine honors is not a policy

A team that can answer these seven questions about two candidate systems does not need the labels at all. A team that cannot answer them has not finished evaluating, whatever the products are called.

Why several structures coexist anyway

Given the costs of the two-tier arrangement, the reasonable question is why so many organizations still run more than one structure. In practice the reasons are rarely about which architecture is better in the abstract.

• What already exists has users. The warehouse holds the definitions finance signs off on and the reports the board reads. Moving it means re-validating every number, which is expensive and invisible when it goes well.
• The workloads genuinely differ. Hundreds of concurrent small BI queries and a handful of jobs scanning terabytes for model training put different pressure on an engine. One system can serve both; whether it serves both well at your volumes is a measurement, not a belief.
• Ownership follows the systems. Once one team runs the warehouse and another runs the lake, consolidation is an organizational change as much as a technical one, and it will be resisted for reasons that have nothing to do with storage formats.
• Some data cannot move. Residency rules, contractual restrictions, and latency to a source system all pin data in place regardless of the target architecture.
• Small data does not need any of this. A dataset that fits on one machine is often better served by single-node analytics than by either architecture, and treating every dataset as a platform problem is its own kind of cost.

Coexistence is therefore a normal end state rather than a transitional embarrassment. What makes it expensive is leaving the boundary undefined. If a dataset is authoritative in one place and copied to another, the copy needs a stated direction, a stated freshness expectation, and a way to check that the two agree. The mechanics of one common boundary are covered in external tables and the lake/warehouse boundary, and the layering convention many teams use inside the lake is medallion architecture.

Deciding for one dataset

The argument in the opening is easier to settle per dataset than in general. These questions, in this order, usually resolve it.

• Who consumes it, and through what interface? SQL from many concurrent users, file reads from training jobs, or both. Both is the case that argues hardest for a single open-format copy — with the caveat that reads the files has to mean reads the table, through a client that understands the table format, not a directory listing of Parquet.
• How fresh must it be, and who notices when it is not? Put the number in hours or minutes. A second hop costs freshness, and data freshness is the property most often assumed rather than stated.
• What is one unit of publication, and what happens when it runs twice? This matters more than whether rows are ever updated. A load that writes forty files has to become visible as one unit, and a rerun has to replace rather than duplicate, which needs a table layer even for data that only ever appends. A single-file append can work without a table layer, provided the file becomes visible atomically and a rerun cannot leave two copies of it — which usually means that a rerun of the same logical batch writes to the same target key and replaces it, rather than generating a fresh name each attempt and accumulating copies.
• How many writers, and do they overlap? One scheduled writer is a different problem from a stream and a backfill touching the same partition.
• Are there updates, deletes, or erasure obligations? These push toward a table layer, though it is not the only route: partition-level overwrite handles some update patterns without one. Erasure needs more than the delete, as described above — snapshot expiry, file rewriting, and a retention policy that fits the deadline.
• What is the cost of a second copy, including the work of keeping it right? Storage is the cheap part. The pipeline, the reconciliation, and the argument about which number is correct are the rest of it.

A dataset read by both audiences, published in multi-file batches by more than one writer, and expensive to duplicate is the case the lakehouse pattern was designed for. A curated, low-volume financial dataset read only by BI, already correct in a warehouse, is not worth moving to prove a point about architecture.

Architecture choices and vendor claims

The 2021 Lakehouse paper by Michael Armbrust, Ali Ghodsi, Reynold Xin, and Matei Zaharia, who had Databricks affiliations, advocates replacing the warehouse architecture and reports that the company’s SQL engine outperformed leading cloud warehouses on a benchmark. That forecast and competitive result need to be weighed against the workloads, configurations, and constraints of the systems being evaluated.

The underlying mechanisms answer more specific design questions. Object stores do not provide atomic multi-file table updates; Delta and Iceberg add metadata that identifies the files in a table version. Min-max statistics let engines skip files. Those mechanisms explain transactional publication and query efficiency, but they do not establish that one architecture will replace every other design or that a benchmark ranking applies to a different workload.

References

• Michael Armbrust, Ali Ghodsi, Reynold Xin, and Matei Zaharia, Lakehouse: A New Generation of Open Platforms that Unify Data Warehousing and Advanced Analytics (CIDR 2021, PDF)
• James Dixon, Pentaho, Hadoop, and Data Lakes (2010)
• Snowflake Documentation, Key Concepts and Architecture
• Delta Lake, Transaction Log Protocol
• Apache Iceberg, Table Spec
• Delta Lake Documentation, What are deletion vectors?
• Amazon Athena: Data protection
• Snowflake: Data storage considerations