Lambda Architecture

The Lambda architecture is a design for data systems that need both low-latency results and complete, correctable ones. It sends the same incoming data down two paths: a batch path that periodically recomputes results over all the data, and a streaming path that updates results continuously as new data arrives. Queries combine the two, so recent activity comes from the streaming side and everything older from the batch side.

(The name is unrelated to AWS Lambda, the serverless compute service.)

The problem it was meant to solve

Jay Kreps, writing in 2014, credited Nathan Marz with describing the idea in a popular blog post, and summarized how it works: an immutable sequence of records is fed into a batch system and a stream processing system in parallel, the transformation logic is implemented in both, and the results are stitched together at query time. One argument made for it was that real-time processing is inherently approximate and more lossy than batch, so the batch path should be the reliable answer; Kreps disputed that argument. The argument he did accept was reprocessing — code changes, and results computed with old code have to be recomputed — and he credited the architecture with highlighting that problem.

The cost: two implementations of one answer

Kreps located the cost precisely. Because the same transformation logic lives in two systems, both have to be kept producing the same answer, and in his words, “maintaining code that needs to produce the same result in two complex distributed systems is exactly as painful as it seems.”

The pain shows up in concrete ways:

• A business rule is changed in the batch code and not in the streaming code, and today’s figures disagree with yesterday’s.
• The two frameworks handle time zones, late data, or deduplication differently, so the same input gives slightly different answers.
• Two systems have to be operated, monitored, upgraded, and understood by whoever is on call.
• Every discrepancy prompts the question of which path is right, and the answer is not always “batch.”

What replaced it, and what remains

Kreps proposed a single streaming path that reprocesses history by replaying a retained log, suggesting — while doubting it merited a Greek letter — that it might be called the Kappa architecture. Where a processing engine can run the same code in both batch and streaming modes, another option is to keep one implementation and run it on different schedules. The term remains in current guidance — Microsoft’s architecture material, for example, still refers to “Lambda hot path” designs for streaming needs.

The durable lesson is not that two paths are always wrong. Different consumers can legitimately need different latencies. The mistake is two independently maintained definitions of the same result. Where two execution paths remain, they should share one definition of the logic, and a regular comparison of their outputs should catch drift before a consumer does.

The detailed comparison of batch, micro-batch, and streaming designs is in batch, micro-batch, and streaming, and how the choice of latency fits into broader integration decisions is worked through in Moving Data Between Systems: Copy or Call, Transform Where, and How Often.

References: Jay Kreps, Questioning the Lambda Architecture (O’Reilly Radar, 2014); Azure Architecture Center, Extract, transform, load (ETL).