Airflow Timetable

The schedule needs a time contract

An Airflow timetable defines the timing information used to schedule a DAG, including the next eligible run and its data interval. It supplies scheduling information; workers execute tasks later when their dependencies and capacity permit. A timetable does not query the source to prove that all expected events have arrived.

Trigger time and processing interval

With CronDataIntervalTimetable("0 0 * * *", timezone="UTC"), October 2 midnight closes the October 1–2 interval. Its logical date is October 1 midnight. A CronTriggerTimetable using the same cron expression, without an interval argument, instead represents a trigger at October 2 midnight with equal interval boundaries. Choosing the same cron expression therefore does not establish the same processing range. Specify the timetable explicitly when that distinction matters.

Use the assigned boundaries

For a report covering October 1, filter start <= event_time < end. An event exactly at October 2 midnight then belongs to the next report. If a worker starts a day late, deriving the range from its current date would select different records. The report should retain its assigned range and separately decide how to handle late-arriving records.

A local day need not be 24 hours

For CronDataIntervalTimetable("0 0 * * *", timezone="America/New_York"), adjacent local midnights define a calendar day. The March 10, 2024 interval starts at 00:00 EST (05:00 UTC) and ends on March 11 at 00:00 EDT (04:00 UTC): 23 elapsed hours. The November 3 interval runs from 00:00 EDT (04:00 UTC) to November 4 at 00:00 EST (05:00 UTC): 25 hours. The logical date is the interval start; the next local midnight closes it and makes the scheduled run eligible. Actual task execution can be later.

Use those assigned boundaries for the source filter. Adding 24 elapsed hours to the spring boundary reaches 01:00 the next local day and includes an extra hour; doing so in autumn stops at 23:00 and omits an hour. A fixed-duration daily schedule answers a different question from a local-calendar schedule. Airflow’s timezone-aware cron schedules follow daylight-saving changes; daily delta schedules do not preserve local midnight across those changes.

Check the context before publishing

Record the timezone, interval boundaries, logical date, and earliest eligible time for representative runs. Include a delayed run and a daylight-saving boundary when using a local calendar. Manual and asset-driven runs can have different context from scheduled runs. Inspecting adjacent intervals through the timetable API checks the time calculation. Verify scheduler and worker operation separately.

Worked example: Apache Airflow Fundamentals: DAGs, Tasks, and Scheduling.

Further reading


Discover more from Insightful Data Lab

Subscribe to get the latest posts sent to your email.

Similar Posts

Questions, corrections, or additional insights?

This site uses Akismet to reduce spam. Learn how your comment data is processed.