Running Pipelines in the Right Order: Schedules, Dependencies, and Failure

The daily load is scheduled for 07:00 and the report is due at 08:00. On Tuesday the report is empty. The load ran at 07:00 exactly as configured, finished in four minutes, and reported success — but the file it reads arrived at 07:20, because the upstream system had a slow night. Nothing was broken. The schedule encoded an assumption that stopped being true, and the pipeline had no way to notice. The example is invented; the pattern is the most ordinary failure in scheduled data work.

This article is about the layer that decides what runs, when, and after what — how work is triggered, how dependencies are expressed, what happens when a step fails, and what refilling history does to everything else. Product behavior below was checked against documentation in September 2026, at the versions named.

A scheduler starts things; an orchestrator finishes them

Cron starts a process at a time. That is useful and it is not orchestration. The difference is what happens after the process starts, and it is worth naming because teams often adopt an orchestrator and keep using it as a fancier cron.

Capability	Why a schedule alone is not enough
Dependencies	Step B must run after A succeeded, not at A’s start time plus a guess
State per run	Knowing which periods have been processed is what makes reruns and gap-filling possible
Retries with limits	A transient failure should be retried a bounded number of times, not manually or forever
Concurrency control	Ten catch-up runs starting at once can overwhelm the source they read
Visibility	Someone has to be able to answer “did yesterday’s 14:00 window run, and what did it do?”

It is equally important to know what the orchestrator does not do. Airflow’s documentation states the boundary plainly: the DAG “doesn’t care about what is happening inside the tasks; it is merely concerned with how to execute them.” Order and status are its business. Whether the numbers are right is not, which is why a green run and correct data are different claims.

Three ways to decide that work should start

Azure Data Factory’s documentation lists exactly three trigger types, and they map to the three general options any orchestrator offers: a wall-clock schedule, a stateful interval, and an event.

	Wall-clock schedule	Stateful interval (tumbling window)	Event
Fires	At the configured times	Once per fixed, non-overlapping interval, keeping state	When something happens — a file lands, a message arrives
Knows about past periods	No: per ADF’s comparison, backfill is not supported and runs only happen from now forward	Yes: runs can be scheduled for windows in the past	Not applicable
Retries the pipeline	Not supported at the trigger level in ADF	Supported through a retry policy in the trigger definition	Depends on the consumer
Relationship	Many-to-many with pipelines	One trigger, one pipeline	Many-to-many
Completion	ADF describes it as “fire and forget” — the trigger run is successful once a pipeline run has started	The trigger run waits for the pipeline run and reflects its state	Depends on the consumer

That last row is the one that surprises people. A schedule trigger that reports success tells you a run started. If you are monitoring trigger status to know whether last night’s load worked, you may be watching a light that was never wired to the thing you care about.

Airflow expresses the stateful version through the data interval: each run is assigned the time range it operates on, the logical date marks “the start of the data interval, not when the Dag is actually executed,” and a run is normally scheduled after its interval has ended so that the period’s data can be collected in full. ADF exposes the same idea as WindowStart and WindowEnd variables passed into the pipeline. Either way, the important habit is to parameterize the work by the interval it covers rather than by “now,” because that is what makes the same run repeatable later.

Waiting for readiness instead of assuming it

Back to the empty report. Three ways out, with different costs.

• Wait for the condition. A sensor is a task whose job is to wait for a file, a partition, or an external state. Airflow documents two modes: the default poke mode, where the sensor “takes up a worker slot for its entire runtime,” and reschedule mode, which takes a slot only while checking and sleeps in between — plus deferrable operators that free the slot entirely while waiting. Every sensor needs a timeout; a sensor that waits forever is an outage that looks like patience. A sensor without a deadline is a job that waits forever and reports nothing, holding a worker while it does; give every sensor a timeout and decide what should happen when it expires — fail loudly, or proceed with the gap recorded.
• Let the arrival start the work. An event trigger — ADF’s storage event trigger fires on a blob being created or deleted — removes the guess entirely. The cost is that the producer’s behavior now drives your schedule, including when it produces something unexpected.
• Declare the data dependency. Modern orchestrators can schedule a job when the dataset it consumes has been updated, rather than at a time chosen to be safely after. This makes the dependency visible in the tool instead of living in a delay someone tuned once.

Adding a fixed delay — “run at 07:30 instead” — is the fourth option, and it is a guess that will be wrong in both directions: too early on a bad night, and an idle half hour on every good one. It is acceptable as an interim measure with an expiry date, which is a different thing from a design.

Expressing dependencies

Inside one pipeline, dependencies are the graph: a task has upstream tasks it waits for and downstream tasks that wait for it. What is less obvious is that “waits for” needs a definition when things go wrong, and Airflow makes that explicit with trigger rules. The default is all_success — every upstream task succeeded — and the alternatives include all_done (they finished, however they finished), one_failed, none_failed, and always.

Those are not exotic settings. A cleanup step that must release a lock belongs on all_done; a notification of partial failure belongs on one_failed; a publish step belongs on the default, because publishing after a failed upstream is exactly the accident you are trying to prevent.

The cleanup case carries a trap worth knowing before you use it, because it can hide the very failure this article is about. A run’s final state is decided by its leaf tasks — the ones with nothing downstream — and Airflow’s documentation warns plainly: “if you have a leaf task with trigger rule ‘all_done’, it will be executed regardless of the states of the rest of the tasks and if it will succeed, then the whole Dag Run will also be marked as success, even if something failed in the middle.” A lock-release task placed at the end of the graph is exactly that shape. The pipeline reports green and the data is wrong.

So design the cleanup and the final state as two separate things — and note that simply putting another task after the cleanup does not work, because a trigger rule looks at that task’s direct parents. A watcher whose only parent is a successful cleanup sees success and never fires.

Airflow documents the working shape as the watcher pattern, and the two requirements are easy to get wrong. The watcher needs one_failed so that “it will be triggered when any task fails and thus fail the whole Dag Run” — and it has to be downstream of every task you want it to notice, since “we need to make it dependent on all of them separately.” It must also actually raise: “the watcher task will be executed and fail making the Dag Run fail too.” A watcher that logs a message and exits cleanly leaves the run green.

Where the orchestrator has a setup/teardown construct, that is the cleaner route, with one detail to check: whether a failing teardown is configured to fail the run, which is a separate setting rather than the default. Either way, test it the only way that settles it — fail a middle task on purpose and look at the run’s state.

Across pipelines, the same question arrives without the convenience of a single graph. ADF’s tumbling window triggers can depend on other tumbling window triggers, and on themselves: a self-dependency makes each window wait for the previous one, which serializes runs of the same pipeline so that a slow window cannot be overtaken by the next. That pattern is worth knowing generally — any pipeline that accumulates state, or writes to the same partition it reads, wants its runs serialized rather than overlapping.

What happens when something fails

Retries, and what they assume

Retry configuration is the first thing anyone turns on and the least examined. ADF’s tumbling window trigger takes a retry policy whose count defaults to 0 and whose interval defaults to 30 seconds, and the service additionally retries automatically on concurrency, server, and throttling errors. Airflow sets retries per task instance. In both cases the mechanism is trivial and the assumption underneath is not:

• The task must be safe to run again. A retry after a partial write repeats whatever the first attempt already did. This is the idempotency requirement that incremental pipelines are built around, and it is the reason “add retries” is sometimes the wrong first move.
• The failure must be transient. Retrying a malformed-input failure three times produces the same error three times, later. Distinguishing retryable from permanent is a code decision, not a configuration one.
• The retry must not amplify the problem. Immediate retries against a struggling source add load exactly when it is least available; spacing them with exponential backoff and jitter is what keeps a slow dependency from becoming a failed one.

Timeouts, propagation, and where to point the alert

A task with no timeout can hang until someone notices, which in practice means until the report is missing. Timeouts belong at the task level, and a run-level bound is worth setting too: past some point, finishing late is worse than not finishing, because a load that lands at noon can collide with the next scheduled run.

Propagation deserves a deliberate answer rather than the default. Should a failed regional load stop the global aggregate, or should the aggregate run with a note that one region is missing? Both are defensible; only one of them is usually chosen on purpose. And alerts should fire where a human can act — a single notification naming the failed task and the affected data period beats twelve downstream failures that all say “upstream failed.”

Whatever is chosen, the status the orchestrator reports is about execution. Correctness needs its own assertions, placed between loading and publishing, as worked through in The Job Succeeded and the Numbers Are Wrong.

Refilling history without taking down the source

Because a stateful scheduler knows which intervals have run, it can run the ones that have not. Airflow calls this catchup, and in current versions it is off by default — the documentation notes scheduler.catchup_by_default=False, so runs that were missed since the last data interval are not created automatically when a DAG is activated. Turning it on for a DAG whose start date is a year in the past creates a year of runs.

ADF documents the arithmetic for its tumbling window trigger explicitly: if the start time is in the past, the trigger generates M = (current time − trigger start time) / window size backfill runs, in parallel up to the concurrency limit, oldest interval first. With a required maxConcurrency between 1 and 50, backfilling a day of hourly windows at concurrency 10 runs the first ten windows, then the next ten. The documentation adds practical advice worth heeding — for a long period, do an initial historical load instead.

The failure mode to picture before enabling any of this: a hundred queued runs starting together, each opening connections to the same operational database that is also serving customers. The concurrency limit is not a tuning parameter in that scenario; it is the difference between filling a gap and causing an incident. See data backfill for what the reprocessing itself has to satisfy.

Where the orchestrator’s responsibility ends

The orchestrator provides	Someone else still owns
Triggering, ordering, retries, run history	Whether the transformation logic is correct
Task status and duration	Whether the data that was loaded is complete and accurate
Concurrency limits it is told to enforce	Knowing what the source can absorb
Alerting on execution failures	Deciding who is called, and what they should do
A place to see dependencies	Keeping declared dependencies matched to real ones

That right-hand column is where most “orchestration problems” actually live. A dependency that exists in the data but not in the graph — a job reading a table that another job writes, with no declared link — will work until timings shift, and then fail intermittently in a way that looks like flakiness. Keeping the declared graph honest is ongoing work, and it is the part no tool does for you.

Product choice matters less than these boundaries, but it is not nothing: managed services reduce the operating burden and constrain what you can express, while a self-run orchestrator gives more control and hands you its availability. Note also that the documentation for a product can point at its own successor — Azure Data Factory’s pages currently direct new data integration work to Data Factory in Microsoft Fabric — which is the kind of statement to check for whichever tool you are adopting.

References

Documentation was checked in September 2026 at the versions named. Defaults such as catchup behavior and retry policies change between versions; confirm them for the version you run.

• Apache Airflow 3.3.2 documentation, Dags, Dag Runs, and Sensors
• Microsoft Learn, Pipeline execution and triggers (Azure Data Factory)
• Microsoft Learn, Create tumbling window triggers (Azure Data Factory)