Sensor
Instead of guessing that the input has landed, a sensor waits for it. What it costs is a worker slot, some polling, and a deadline you must remember to set.
-
-
Data Interval
The 07:00 run usually processes yesterday, not this morning. Naming the interval a run covers is what makes a rerun mean the same thing as the original.
-
Workflow Orchestration
An orchestrator is not a fancier cron. It keeps state per run, enforces order, retries within limits — and still says nothing about whether the data is right.
-
Pipeline Orchestration and Scheduling
Running at 07:00 is not the same as running when the data is there. How orchestrators decide what runs, in what order, and what happens when a step fails.
-
Deferrable Operator and Sensor
Release workers during supported waiting, distinguish deferral from reschedule mode, and account for triggerer capacity and pool policy.
-
Airflow Pool
Limit participating Airflow tasks with weighted pool slots, distinguish concurrency from request rate, and plan how deferred tasks count.
-
Airflow Timetable
Learn how Airflow timetables define run timing and data intervals, compare trigger and interval schedules, and check time boundaries before deployment.
-
Airflow TaskFlow API
Separate TaskFlow graph construction from task execution, follow returned values through XCom, and recognize the limits of ordinary Python-looking calls.
-
Airflow Catchup
Understand how Airflow catchup selects missing intervals, compare it with retries and backfills, and avoid confusing scheduled work with complete data.
-
Trigger Rule
Choose which upstream states allow a task to run, distinguish all_success from all_done, and check how leaf tasks affect the overall run status.
-
Airflow XCom
Pass small values or versioned data references between Airflow tasks, and separate metadata exchange from durable shared storage.
-
Data Interval and Logical Date
Use a run’s intended data interval instead of wall-clock time, check boundaries, and account for timetable-specific logical dates.
-
Task Instance and Retry
Track an Airflow task within a run, distinguish attempts from executions, and make repeated destination effects safe before adding retries.
-
Airflow DAG and DAG Run
Separate an Airflow workflow definition from each execution, trace task dependencies, and understand why different runs can overlap.