Batch, Micro-Batch, and Streaming: Choosing How Fresh Data Needs to Be

In plain terms

Data can be delivered daily, periodically in small batches, or continuously. These choices affect scheduling, state, recovery, and when results become useful. They do not imply fixed cost ratios or separate technology stacks: a streaming engine may process micro-batches, and a batch pipeline may retain state between runs.

Ask what someone will do with fresher data, by when, and what happens if it is late or wrong. A morning report may need a scheduled job; an operational dashboard or automated response may need updates within seconds. Source delivery, available compute, and the consumer’s response time constrain what is achievable.

This article compares the modes and develops a decision method. One possible hybrid continuously collects events and processes retained data on a schedule, with a faster path for selected consumers. The lab compares three implementations on the same generated events. Their errors follow from the chosen late-data and recovery policies, not from unavoidable defects in a mode.

Three words that get confused

• Latency is how long one record takes to travel from creation to being visible. Measured in seconds, minutes, or hours.
• Throughput is how many records the system can move per second. A nightly batch can have enormous throughput and terrible latency.
• Freshness describes the age of the data available at observation time. Define the reference: last successful source sync, latest event time, or a verified completeness boundary. A dashboard last updated at 06:00 is eight hours since refresh at 14:00; its data may be older. A continuously updated dashboard does not become week-old merely because it is read weekly.

Measure freshness, event-to-visible latency, and time to action separately. A dashboard opened once a day may only need a scheduled refresh before review, but automated consumers or a requirement to show the latest state when opened can justify more frequent updates.

Where the time actually goes

event-to-visible latency = source publication + transport/queue wait
                         + scheduling wait + processing + serving publication
event-to-action latency  = event-to-visible latency + consumer response time

Example: 2 min source delay + 4 min schedule wait + 1 min processing/serving
       = 7 min until visible. A further 3 min to act gives 10 min until action.

Specify the start and end points of the latency budget and measure the distribution, including queueing and failures. A five-minute trigger does not guarantee five-minute event-to-visible latency. Check source timeliness and publication delays as well as the job’s runtime.

The three modes

Batch

Process a bounded input, such as a file set or a specified interval. The job can finish that input without knowing whether all real-world events for the period have arrived. Reruns require retained inputs, suitable code and reference data, and replay-safe writes. Incremental batch jobs may also retain offsets, previously processed IDs, and aggregate state.

A bounded job is often convenient for a periodic report or backfill: fixtures can specify input and expected output, and a failed partition can be rebuilt. Use it when the full schedule and runtime meet the requirement. Report grain is separate from refresh cadence: a daily revenue total can be updated every minute.

Micro-batch

Micro-batching repeatedly processes small bounded increments. An hourly scheduled incremental job is one example; streaming engines can also execute much shorter micro-batches. Existing SQL may be reusable, but offsets, idempotent output, and late-data handling still need design. Spark’s Structured Streaming guide distinguishes its default micro-batch execution from continuous processing.

The useful interval depends on startup overhead, available capacity, and workload; there is no universal one-minute floor. An event created at 09:59:58 and received at 10:00:01 belongs to the earlier event-time window if that is the chosen grouping. It becomes an error only when the implementation ignores it or groups it by the wrong clock.

Streaming

Streaming processes an ongoing input without requiring an overall end. Individual operations can be stateless, while windowed sums and deduplication keep state. Event time is when an event happened, ingestion time is when it was received, and processing time is when an operator processes it; backlog can separate the latter two. A watermark expresses progress in event time. Window emission, finalization, and handling of later records depend on the engine and output policy.

For example, if the latest event time seen is 09:08 and the watermark delay is three minutes, the toy watermark is 09:05. It closes the five-minute window [09:00, 09:05), whose start is included and end excluded. This is an event-time threshold, not a timer that waits three wall-clock minutes after every arrival. No newer events means this watermark cannot advance.

The lab uses non-overlapping five-minute tumbling windows. A ten-minute sliding window with a one-minute slide defines overlapping groups, so one event can contribute to several outputs. The slide sets window boundaries; triggers and output policy determine when results are actually emitted. A session window groups activity by an inactivity gap, usually per user or another key; a late event can connect two sessions if the policy permits revision. Choose the grouping before choosing when to publish it.

Streaming can support automated decisions and frequently refreshed reports when their latency and operating requirements justify it. Early results may be revised, or a system may delay final output until an agreed completeness condition is met. Batch and streaming both need explicit correctness criteria.

A maximum-timestamp watermark is sensitive to bad clocks. A record accidentally dated tomorrow can move it past today’s open windows, after which valid arrivals can be dropped. Validate timestamp units, timezone and plausible future skew before updating progress; retain rejected input for review. In a multi-input job, a common policy is the minimum of active input watermarks. An idle input can stall it; excluding that input restores progress but does not make its later backlog timely. The exact rules depend on the engine.

Comparing processing implementations

The main comparison is how each implementation bounds input, persists progress, and corrects earlier outputs. A batch deadline can require overnight response; a streaming pipeline can be non-critical. Assign on-call coverage from the service objective, not from the processing label.

Finding the real requirement

Requirements for freshness are rarely stated precisely at first, because “real-time” sounds like a quality and “daily” sounds like a compromise. A short conversation gets the real requirement. Ask, in this order:

1. “What decision or action does this data drive?” A report may support an immediate action or a weekly review. Name that action before selecting a mode.
2. “How quickly does that action have to happen after the event?” Not “how quickly would you like the data”, but how quickly the action must happen. Fraud: before the payment clears, seconds. A stock-out: before the next order, minutes. A marketing segment: before tomorrow’s send, hours.
3. “What does it cost when the data is one hour older than that?” Record the estimated impact, its uncertainty, and any non-monetary requirement.
4. “What does a wrong answer cost?” A result may be provisional until the chosen close condition holds, in any mode. Early counts get revised as late events arrive, or stay short if the late events are dropped. If a frozen report is needed, define a close boundary and a controlled correction process. Batch does not guarantee that late data will stop arriving.
5. “Who looks at it, and when?” Distinguish when the output is refreshed from when a person acts on it.

Write the answer as a latency budget in the non-functional requirements (the architecture article’s NFR page): “fraud signal available to the decision service within 60 s of the event, 99% of the time” or “daily marts complete by 06:00 UTC”. A budget is testable; “real-time” is not.

Illustrative requirements by domain

These rows illustrate possible requirements, not a measured distribution of company workloads. Decide each path from its deadline, accuracy policy, and operating cost. Shared infrastructure can serve several consumers without forcing them to share one job or failure domain.

Hybrid architectures: what has been tried

Lambda: two paths, one answer (in theory)

              ┌── batch layer: retained-history view (scheduled recomputation) ──┐
events ──────►│                                                                          ├──► serving layer merges both
              └── speed layer: recent-input view (continuous updates) ───────────┘

Lambda combines a batch-derived view of retained history with a speed view of recent input. The cutover boundary must prevent gaps and double counting. Maintaining equivalent logic across paths creates testing and operational work; shared definitions and reconciliation can reduce divergence. A batch layer need not recompute everything nightly, and the speed view is not inherently approximate.

Kappa: one path, replay for everything

Kappa uses a stream-processing path for both ongoing computation and replay. Reprocessing requires the relevant retained input, compatible schemas, code and reference data, sufficient capacity, and a plan to replace the old output. One path reduces duplicated implementations but does not guarantee identical results across changed versions. Required history may live in a replayable log or other retained storage; assess the replay mechanism before relying on it.

A hybrid: continuous collection and scheduled transforms

                     streaming INGESTION                          batch PROCESSING
events ──► Kafka ──► continuous writer ──► lake table (Iceberg)  ──► hourly / daily SQL (dbt) ──► core, marts
                          │                 (append-only, minutes old)
                          └──► narrow streaming lane: one stream job ──► fraud / alerts / real-time store

Collection cadence and transformation cadence can differ. A continuous writer can preserve events for scheduled SQL and a faster consumer. Reliability still depends on producer acknowledgements, durable buffering, retention, retries, and capacity. Hourly scheduling alone does not guarantee hourly freshness. If two paths compute related metrics, align event-time rules, deduplication, late corrections, and versions, then reconcile results; writing into the same core layer is not enough.

Correctness pitfalls, per mode

Batch and micro-batch: the boundary

A job at 02:00 can use only the input available by its cutoff. An order placed at 23:58 and captured at 00:03 legitimately has different order and capture dates; choose the date matching the metric. A phone event that arrives twelve hours late still belongs to its earlier event-time window if that is the metric’s rule. Ignoring such arrivals leaves the earlier result incomplete.

A lookback rebuilds recent event-time partitions using newly arrived input. Choose its duration from observed delay and correction distributions, the accepted residual error, and rebuild cost. Data outside the lookback needs another route, such as a targeted backfill or correction ledger. Reruns should replace or upsert the affected result rather than append duplicates. Tell consumers which versions are provisional and how closed reports are corrected.

Streaming: time, order, and duplicates

• Out-of-order and late events. Two events from two phones can arrive in the opposite order to how they happened. Reaching 09:05 on the clock does not establish that the window “09:00–09:05” is complete: a 09:04 event may still be in transit. A system can choose to close then, but must accept the resulting exclusions or provide corrections. Watermarks track event-time progress; triggers and allowed-lateness rules determine emissions, revisions, and eventual state cleanup. The toy below chooses immediate finalization at its watermark, but other policies can accept late revisions. Every choice is a trade between latency and completeness, and it must be made explicitly.
• Duplicates. Producers retry (the networks article). The same event arrives twice. Deduplication needs the system to remember recent event IDs, which is state, for as long as duplicates can be apart.
• State growth. “Last event per user” needs one entry per retained user. New users increase state unless a retention policy removes old entries. Plan storage and checkpoint capacity; legitimate retained state is not itself a memory leak.
• Replay safety. Recovery replays from a checkpoint, so some records are processed twice. The output sink must tolerate that (idempotent writes, transactional sinks) or the recovery itself corrupts the result.

Checkpointing and exactly-once effects are different promises. If an external counter is incremented and the process fails before saving its input position, replay can increment it again. Saving the position first can instead lose the effect after a failure. Coordinate input progress, operator state, and committed output, or use a replay-safe sink operation. At-least-once delivery can repeat records; an exactly-once claim must name its scope and supported source/sink. Engine recovery does not automatically deduplicate repeated business events or external notifications.

Test late input, duplicate delivery, state growth, and recovery against the chosen policy. Expiring IDs limits state but also limits how old a duplicate can be and still be recognized.

The lab treats an event ID as an immutable event: every redelivery has the same timestamp and amount. A revised amount under the same ID is a correction, not an identical retry. The batch reference keeps the first value, the micro-batch dictionary keeps the last available value, and the stream ignores an ID it has seen. They need not agree for conflicting payloads. Validate this invariant, or define version ordering and correction/retraction rules before comparing those implementations.

Estimating cost from the workload

Capacity sets a lower bound on freshness. At a steady arrival rate of 1,000 records/s and processing capacity of 800 records/s, backlog grows by 200 records/s: 120,000 records in ten minutes. If capacity then rises to 1,500 records/s while arrivals continue at 1,000, clearing that backlog takes at least 240 seconds under this constant-rate model. Backpressure slows upstream work when downstream cannot keep up; buffering buys time but has finite retention. Monitor lag, the oldest pending input, hot keys, and checkpoint duration, and budget capacity for replay as well as new traffic.

Compare costs with the same event volume, transformations, correctness target, and recovery requirement. Estimate billed compute time and capacity, storage and scans, transport, state/checkpoints, and operational effort. This article has no measured basis for a universal cost multiplier.

Frequent incremental processing may avoid expensive full rescans, while idle resources and large state can increase cost. Neither mode is automatically cheaper. Choose the least costly design that meets the agreed service and correctness requirements.

Worked example: assigning modes

Use the fictional subscription company from the architecture article. These are assumed answers to confirm with the owners:

In this fictional example, several consumers can share ingestion and metric definitions. Test whether the operations dashboard meets its latency budget with incremental scheduling, and whether fraud and ML can share processing without conflicting state, scaling, or failure requirements. Pipeline count is an implementation choice.

Anti-patterns

• Streaming as a default. Built because the technology is interesting. Extra operational work without a confirmed latency benefit.
• Two implementations of one metric. Lambda by accident: a batch revenue and a streaming revenue that disagree. Keep one governed definition; when implementations differ, reconcile their cutoffs, versions, and results.
• Batch with no reprocessing window. Late events permanently miscounted; “yesterday” frozen at 02:00 forever.
• Streaming with no late-data policy. The watermark was left at its default; nobody knows what happens to a five-minute-late event. Find out before production does.
• Micro-batch racing itself. A 5-minute interval whose job takes 7 minutes. Without serialization or safe concurrency, runs can conflict; serial execution instead accumulates backlog. Add capacity, reduce work, or revise the schedule against the latency budget.
• Measuring only job duration. A two-second job can process a day-old backlog. Monitor input age and event-to-visible latency as well as runtime.
• “Real-time” with no budget. Not a requirement until it has a number and a percentile.

Lab

The lab uses Python’s standard library to simulate 2,000 order events in one hour from 09:00, with 60 identical redeliveries. Base delay is 5–29 seconds. Independently for each event, a 5% branch adds 30–119 seconds and a 1% branch adds 600–1,499 seconds; actual sample proportions vary. All timestamps use the same assumed UTC clock. One assumed currency, USD, and floating-point amounts keep the example small; this is not an exact money implementation. Run setup and the batch reference once, then each solution in the same session. Arrival time also serves as processing time here: no queueing or compute duration is modeled.

import copy, random
from collections import defaultdict
from datetime import datetime, timedelta

rng = random.Random(0)
start = datetime(2026, 3, 14, 9, 0)
events = []
for i in range(1, 2001):
    event_time = start + timedelta(seconds=rng.randrange(3600))
    delay = rng.randrange(5, 30)
    roll = rng.random()
    if roll < 0.05:
        delay += rng.randrange(30, 120)
    elif roll < 0.06:
        delay += rng.randrange(600, 1500)
    events.append({"event_id": f"E{i}", "event_time": event_time,
                   "arrival_time": event_time + timedelta(seconds=delay), "amount": round(rng.uniform(5, 100), 2)})
duplicates = [dict(e, arrival_time=e["arrival_time"] + timedelta(seconds=rng.randrange(1, 60)))
              for e in rng.sample(events, 60)]
stream = sorted(events + duplicates, key=lambda e: e["arrival_time"])

def window(event_time):
    return int((event_time - start).total_seconds() // 300)

delays = [(e["arrival_time"] - e["event_time"]).total_seconds() for e in events]
print(len(stream), "records,", len(events), "distinct events,", sum(d > 600 for d in delays), "more than ten minutes late")
# 2060 records, 2000 distinct events, 18 more than ten minutes late
latest_seen, out_of_order = start, 0
for e in stream:
    out_of_order += e["event_time"] < latest_seen
    latest_seen = max(latest_seen, e["event_time"])
print(out_of_order, "records arrived after an event that happened later")
# 1362 records arrived after an event that happened later

Two-thirds of the records arrive after some event that happened later than they did. Out-of-order is not the exception in this stream. It is the normal condition, and it is what every mode below has to cope with.

1. Batch reference. Wait until every generated delivery is available, then deduplicate by event ID and sum by event-time window. The finite generator tells us what complete input means; 10:00 alone does not. This reference tests the other policies against retained input, not against independent business truth.

seen, truth = set(), defaultdict(float)
for e in stream:
    if e["event_id"] not in seen:
        seen.add(e["event_id"])
        truth[window(e["event_time"])] += e["amount"]
truth = {w: round(v, 2) for w, v in sorted(truth.items())}
print(truth)
# {0: 7292.95, 1: 9173.65, 2: 10024.32, 3: 8137.82, 4: 8367.84, 5: 10284.0, 6: 8908.26, 7: 8772.3, 8: 7712.28, 9: 9736.43, 10: 7857.93, 11: 9410.66}
print(len(seen), "events counted once")
# 2000 events counted once
print("all deliveries available:", max(e["arrival_time"] for e in stream).strftime("%H:%M:%S"))
# all deliveries available: 10:21:31

Window 0 is [09:00, 09:05), window 1 is [09:05, 09:10), and so on. Deduplication counts 2,000 generated events once. The printed last-arrival time gives the earliest point at which this full reference input is available, excluding execution time. In production, define a source-completeness check and a policy for later corrections rather than assuming the hour boundary closes the data.

2. Micro-batch. Run every five minutes over whatever has arrived so far and recompute the newest closed window. Then add a lookback that also recomputes the previous 1, 3, and 6 windows on each run. For each setting, count the windows that differ from the batch answer and the revenue missing in total.

def micro_batch(lookback):
    """Run every five minutes for ninety minutes; each run recomputes the newest closed window and `lookback` windows before it."""
    if type(lookback) is not int or lookback < 0:
        raise ValueError("lookback must be a non-negative integer")
    results = {}
    for run in range(1, 19):
        cutoff = start + timedelta(minutes=5 * run)
        arrived = [e for e in stream if e["arrival_time"] < cutoff]
        latest = run - 1
        for w in range(max(0, latest - lookback), min(latest, 11) + 1):
            amounts = {e["event_id"]: e["amount"] for e in arrived if window(e["event_time"]) == w}
            results[w] = round(sum(amounts.values()), 2)
    return results

for lookback in (0, 1, 3, 6):
    result = micro_batch(lookback)
    wrong = [w for w in truth if result.get(w) != truth[w]]
    print(f"lookback {lookback}: {len(wrong):2d} windows wrong, revenue short by {sum(truth.values()) - sum(result.values()):.2f}")
# lookback 0: 12 windows wrong, revenue short by 7823.07
# lookback 1: 10 windows wrong, revenue short by 990.44
# lookback 3:  8 windows wrong, revenue short by 515.66
# lookback 6:  0 windows wrong, revenue short by 0.00

3. Streaming. Process unique events in arrival order. In this toy, the watermark is maximum observed event time minus a configured watermark delay. Finalize a window when its end is at or before the watermark, and drop later unique arrivals for it. This also applies when no earlier event from that window was seen. Start with three minutes; compare a fresh restart after record 1,234 with recovery from state and input position saved every 500 records. Then try a fifteen-minute watermark delay.

class Consumer:
    def __init__(self, watermark_delay):
        if watermark_delay < timedelta(0):
            raise ValueError("watermark delay must be non-negative")
        self.watermark_delay = watermark_delay
        self.watermark = start - watermark_delay
        self.totals, self.seen, self.emitted = defaultdict(float), set(), {}
        self.latest_event_time, self.dropped = start, 0

    def process(self, e):
        if e["event_id"] in self.seen:
            return
        self.seen.add(e["event_id"])
        w = window(e["event_time"])
        if start + timedelta(minutes=5 * (w + 1)) <= self.watermark:
            self.dropped += 1
            return
        self.totals[w] += e["amount"]
        self.latest_event_time = max(self.latest_event_time, e["event_time"])
        self.watermark = self.latest_event_time - self.watermark_delay
        for closed in [w for w in self.totals if w not in self.emitted and start + timedelta(minutes=5 * (w + 1)) <= self.watermark]:
            self.emitted[closed] = round(self.totals[closed], 2)

    def results(self):
        # Snapshot for inspection: includes both finalized and still-open windows.
        return {w: self.emitted.get(w, round(t, 2)) for w, t in sorted(self.totals.items())}

def report(name, consumer):
    wrong = [w for w in truth if consumer.results().get(w) != truth[w]]
    return f"{name:28}: {len(wrong):2d} windows wrong, {consumer.dropped:2d} late events dropped, {len(consumer.seen)} ids in state, {len(consumer.emitted)} windows finalized"

steady = Consumer(timedelta(minutes=3))
for e in stream:
    steady.process(e)
print(report("steady run", steady))
# steady run                  :  9 windows wrong, 17 late events dropped, 2000 ids in state, 11 windows finalized

crash_at = 1234
first_half = Consumer(timedelta(minutes=3))
checkpoint, checkpoint_offset = None, 0
for offset, e in enumerate(stream[:crash_at]):
    if offset % 500 == 0:
        checkpoint, checkpoint_offset = copy.deepcopy(first_half), offset
    first_half.process(e)

restarted_fresh = Consumer(timedelta(minutes=3))
for e in stream[crash_at:]:
    restarted_fresh.process(e)
print(report("crash, restart with no state", restarted_fresh))
# crash, restart with no state: 11 windows wrong, 12 late events dropped, 799 ids in state, 5 windows finalized

restored = copy.deepcopy(checkpoint)
for e in stream[checkpoint_offset:]:
    restored.process(e)
print(report("crash, restore checkpoint", restored))
# crash, restore checkpoint   :  9 windows wrong, 17 late events dropped, 2000 ids in state, 11 windows finalized

patient = Consumer(timedelta(minutes=15))
for e in stream:
    patient.process(e)
print(report("watermark delay 15 minutes", patient))
# watermark delay 15 minutes  :  5 windows wrong,  6 late events dropped, 2000 ids in state, 8 windows finalized
if restored.__dict__ != steady.__dict__:
    raise AssertionError("Checkpoint replay differs from the steady state")

# Probe timestamp validation with separate consumers.
clock_probe = [
    {"event_id": "A", "event_time": start + timedelta(minutes=1),
     "arrival_time": start + timedelta(minutes=2), "amount": 10.0},
    {"event_id": "bad-clock", "event_time": start + timedelta(days=1),
     "arrival_time": start + timedelta(minutes=3), "amount": 20.0},
    {"event_id": "B", "event_time": start + timedelta(minutes=2),
     "arrival_time": start + timedelta(minutes=4), "amount": 30.0},
]
unchecked, checked = Consumer(timedelta(minutes=3)), Consumer(timedelta(minutes=3))
rejected = []
for e in clock_probe:
    unchecked.process(e)
    if e["event_time"] > e["arrival_time"] + timedelta(minutes=1):
        rejected.append(e["event_id"])
    else:
        checked.process(e)
print("unchecked late drops:", unchecked.dropped)
# unchecked late drops: 1
print("checked late drops:", checked.dropped, "rejected:", rejected)
# checked late drops: 0 rejected: ['bad-clock']

A small fault injection shows why timestamp validation belongs before watermark advancement. These are new consumer instances, separate from the previous runs. The one-minute future-skew limit is an illustrative source contract, not a universal threshold. The test assumes the receiver clock is trusted; it does not detect every invalid timestamp. In production, preserve the rejected record and reason, not only its ID.

4. Compare. For each policy, record the windows that differ from the reference, which values are finalized, and what input or event-time progress is needed before publication. Separate simulated timestamps from actual execution performance. Choose a mode for the finance and operations examples and state its correction policy.

5. Write the decision memo. For the subscription company’s six requests, run the five questions and produce the mode table above in your own words. Add the latency budgets to the NFR page from the architecture article’s lab.