Batch Processing

Batch processing operates on a bounded collection of data: the input for a run has a defined extent. It might be a fixed file list, a database snapshot, or records selected up to an explicit cutoff. The input does not need to be large, and the job does not need to run overnight.

Give a run a clear boundary

Suppose a job calculates Monday’s sales using files delivered by 02:00 Tuesday. Its input is finite, so the job can finish. If an additional Monday file arrives at 03:00, that file was outside the original input. Calling the job complete does not establish that every Monday sale has arrived.

To revise the total, the team can rerun with a new input set. It must also decide whether the result replaces the old total or adds a correction. Appending the full recalculated total would count the earlier sales again.

Frequency does not define the model

Apache Beam distinguishes bounded data suitable for batch processing from continuously growing input. A job launched every minute can still process a bounded batch. Some streaming systems also execute internally as a succession of small batches.

For this sales example, saving the input file list and calculation version makes a result easier to explain and reproduce. Batch processing provides a finishing boundary; it does not by itself provide complete inputs, safe reruns, or a correct business date.

Reference: Apache Beam: Basics of the model.