Kappa Architecture

The Kappa architecture is a design in which all data processing runs through a single stream processing path, and recomputation is done by replaying retained input rather than by a separate batch system. Jay Kreps described it in 2014 — suggesting the name while doubting the idea merited a Greek letter — as an alternative to the Lambda architecture, whose two parallel paths required the same logic to be implemented and kept consistent twice.

How reprocessing works

The key question for any streaming-only design is what happens when the logic changes or a bug is found, and past results need to be recomputed. Kreps described the procedure:

1. Keep the full input log, in a system such as Kafka, for as long as you may need to reprocess.
2. When reprocessing is needed, start a second instance of the job with the new code, reading from the beginning of the retained log.
3. Have it write to a new output table.
4. When it has caught up to the present, switch consumers to the new output.
5. Stop the old job and delete the old output.

There is one implementation of the logic, and “batch” becomes simply a stream job that starts further back.

What it depends on

The design is only as good as the conditions that make replay possible and correct.

• Retention. You can only recompute the history you kept. Retaining a long, complete log costs storage, and the retention period effectively sets how far back a correction can reach.
• Replay time. Reprocessing months of events has to finish in acceptable time. Kreps noted the natural response — raise the parallelism of the reprocessing job so it completes quickly — which means capacity for that burst has to exist.
• Reproducible logic. Replay gives the same answer only if the job does not depend on things that have since changed, such as a lookup against a live service or today’s date. Reference data needs to be versioned or captured in the log too.
• Reliable stream semantics. The ordinary streaming obligations still apply. Microsoft’s guidance on streaming reliability lists checkpointing for at-least-once processing and recovery, idempotent transformations, watermarks for late data, and dead-letter handling for records that cannot be processed.
• Two outputs for a while. During reprocessing, old and new outputs coexist, and consumers need a clean way to switch — and to switch back if the new output is wrong.

Where it stands now

The distinction between Lambda and Kappa matters less where a processing engine can run the same code in both batch and streaming modes. What persists from Kappa is the principle: one definition of the logic, with recomputation by replaying retained input. Whether that input is a message log, a table’s change history, or files kept in object storage matters less than whether it is complete and retained long enough. Replay itself is covered in checkpoint and replay, and how latency choices fit into wider integration decisions is worked through in Moving Data Between Systems: Copy or Call, Transform Where, and How Often.

References: Jay Kreps, Questioning the Lambda Architecture (O’Reilly Radar, 2014); Azure Architecture Center, Extract, transform, load (ETL).