Sensor
Instead of guessing that the input has landed, a sensor waits for it. What it costs is a worker slot, some polling, and a deadline you must remember to set.
-
-
Pipeline Orchestration and Scheduling
Running at 07:00 is not the same as running when the data is there. How orchestrators decide what runs, in what order, and what happens when a step fails.
-
Multi-Agent LLM Design Tradeoffs
Splitting one agent into several buys context isolation, parallelism, and separate permissions, and costs tokens, handoff loss, and new failure paths. Measure it against a single-agent baseline.
-
Asset-Oriented Orchestration: Dagster and Prefect
Declare data assets instead of tasks, record what each production run created, map partitions to find what a correction affects, and gate publication with checks.
-
Asset Partition Mapping
Map a downstream asset partition to the upstream partitions it needs, including windows, and keep logical dependencies separate from storage layout.
-
Asset Materialization
Connect an asset’s production event to its physical output, version, and partition, without confusing execution history with durable or complete data.
-
Asset Check
Attach a named test to a data asset, configure whether failure blocks downstream work, and distinguish blocking from rollback or source completeness.
-
Deferrable Operator and Sensor
Release workers during supported waiting, distinguish deferral from reschedule mode, and account for triggerer capacity and pool policy.
-
Airflow Pool
Limit participating Airflow tasks with weighted pool slots, distinguish concurrency from request rate, and plan how deferred tasks count.
-
Airflow Timetable
Learn how Airflow timetables define run timing and data intervals, compare trigger and interval schedules, and check time boundaries before deployment.
-
Airflow TaskFlow API
Separate TaskFlow graph construction from task execution, follow returned values through XCom, and recognize the limits of ordinary Python-looking calls.
-
Airflow Catchup
Understand how Airflow catchup selects missing intervals, compare it with retries and backfills, and avoid confusing scheduled work with complete data.
-
Trigger Rule
Choose which upstream states allow a task to run, distinguish all_success from all_done, and check how leaf tasks affect the overall run status.
-
Airflow XCom
Pass small values or versioned data references between Airflow tasks, and separate metadata exchange from durable shared storage.
-
Data Interval and Logical Date
Use a run’s intended data interval instead of wall-clock time, check boundaries, and account for timetable-specific logical dates.
-
Task Instance and Retry
Track an Airflow task within a run, distinguish attempts from executions, and make repeated destination effects safe before adding retries.
-
Airflow DAG and DAG Run
Separate an Airflow workflow definition from each execution, trace task dependencies, and understand why different runs can overlap.