Data Interval

A data interval is the period of data that one scheduled run is responsible for, as distinct from the moment the run executes.

Airflow assigns a run scheduled by a data-interval timetable such an interval and defines its logical date as “the start of the data interval, not when the Dag is actually executed”; such a run is normally scheduled after its interval has ended, which the documentation explains is “to ensure the run is able to collect all the data within the time period.” Runs of other kinds — triggered by hand, driven by an asset update, or on a timetable that fires at points in time — do not carry the same interval arithmetic, so the range a run covers is defined by its timetable and run type rather than by one universal rule.

Azure Data Factory’s tumbling window trigger expresses the interval idea as fixed-size, non-overlapping, contiguous windows, passing WindowStart and WindowEnd into the pipeline.

Why the distinction matters

A job that filters on “yesterday” computed from the clock targets different data every time it runs. A job that filters on the interval it was handed targets the same slice whenever it runs — this morning, or in three months when someone reprocesses. That fixes what the run is about; it does not by itself fix what the run will produce, because the rows in that slice can change after late records arrive, a reference table moves on, or the transformation code changes. Rerunnable and reproducible are different guarantees: the second additionally needs the input snapshot, the reference data, and the code version pinned, and a rerun made to correct a period is supposed to differ. What the fixed interval makes possible:

• Reruns that mean something. Re-running the window 2026-03-02 00:00–01:00 reprocesses that hour, not the hour in which you clicked the button.
• Filling gaps. Because the scheduler knows which intervals have run, it can create the ones that have not. Airflow calls this catchup, and notes that it is off by default in current versions; ADF’s tumbling window trigger generates the missing windows from its start time, oldest first, within the concurrency limit.
• Partition-aligned writes. When a run’s interval is the whole of one destination partition and the run reads everything that partition should contain, it can replace that partition, which turns a rerun into a replacement rather than an addition. An hourly run writing into a daily partition, or a run that read only incremental changes or a partial delivery, has no complete replacement to write and needs an apply rule instead — and an older rerun must not overwrite a newer result, so publication order or a version has to be controlled as well.

Where it goes wrong

• Reading “now” inside the task. One current_date in the query undoes the arrangement, because the run no longer even targets its interval; removing it is necessary for a repeatable result, not sufficient.
• Boundary and time zone mismatches. The interval is half-open — start inclusive, end exclusive — and the data’s timestamps may be in a different zone than the schedule. A row at exactly midnight belongs to one interval, and it should be the same one on every engine that reads it.
• Confusing the interval with data arrival. Records for an interval can arrive after that interval’s run has completed. ADF’s delay property postpones the start of processing without changing the window’s start time, which helps with small lags; genuinely late data needs a lookback or a reprocessing policy instead.
• Changing the size later. ADF’s documentation notes that a tumbling window trigger’s frequency and interval cannot be edited after publication, which is a reasonable constraint given that reruns and dependencies are evaluated against those boundaries.

How intervals interact with triggers, dependencies, and backfills is worked through in Running Pipelines in the Right Order. For the run objects themselves in one implementation, see Airflow DAG and DAG run.

Further reading

• Apache Airflow documentation, Dag Runs
• Apache Airflow documentation, Timetables
• Microsoft Learn, Create tumbling window triggers