Micro-Batch Processing

Micro-batch processing divides continuing work into bounded increments and runs them repeatedly. The logical input may keep growing while each execution has a finite input range. Small describes the increments relative to the workload; there is no universal row count or duration that separates a micro-batch from a batch.

Suppose a job starts every five minutes. An event waits four minutes for the next run, then spends two minutes in processing and publication. It becomes visible after six minutes even with no source delay. The trigger interval is therefore not an end-to-end latency guarantee. A seven-minute run on one serial worker also cannot sustain one completed run every five minutes at that workload.

Define the input range, commit progress with replay-safe output, and decide how later arrivals revise earlier results. Retrying an increment must not add its entire total twice. Spark Structured Streaming 3.5.6 documents micro-batch as its default execution model and continuous processing separately; that product distinction does not make micro-batching a different business freshness requirement.

Reference: Spark 3.5.6 Structured Streaming.