Workflow Orchestration

Workflow orchestration is the layer that decides which pieces of work run, in what order, under what conditions, and what happens when one of them fails. In data platforms it sits between the schedule someone wants (“refresh the warehouse every morning”) and the engines that do the work (a query, a Spark job, a copy activity).

Three layers that are easy to confuse

A scheduler can start a job; only an orchestrator can say that the aggregate must wait for all three regional loads, that a failed step should be retried twice and then alert, and that yesterday’s 14:00 window is still missing.

What an orchestrator provides

• A dependency graph. Airflow models a workflow as a DAG whose tasks have upstream and downstream relationships, with trigger rules deciding what “ready to run” means when an upstream task fails or is skipped — the default being that all upstream tasks succeeded.
• State for every run. Knowing which periods have run is what makes gap-filling and reruns possible. Azure Data Factory draws this line between its trigger types: a tumbling window trigger retains state and can run windows in the past, while a schedule trigger runs only from now forward.
• Bounded retries. Failures are retried a configured number of times rather than manually or endlessly.
• Concurrency limits, so that catching up on missed periods does not start fifty runs against one database at once.
• An operational record. Which runs happened, how long they took, what failed and when — the artifact an on-call engineer reads at 03:00.

Where its responsibility ends

Airflow’s documentation states the boundary directly: the DAG “doesn’t care about what is happening inside the tasks; it is merely concerned with how to execute them.” An orchestrator reports that a task exited successfully. It does not know whether the query was correct, whether every row arrived, or whether a filter silently dropped a day of data. Treating a green run as evidence of correct data is the most common misreading of this layer, and it is why data assertions belong in the pipeline alongside the orchestration.

A second limit is that the graph only contains the dependencies someone declared. A job that reads a table another job writes, without a declared link between them, works until timings shift — and then fails intermittently in a way that looks like flakiness rather than like the missing edge it is. Keeping declared dependencies matched to real ones is continuing work that no tool does for you.

How triggering, dependencies, retries, and backfills fit together in practice is worked through in Running Pipelines in the Right Order. For one widely used implementation, see Apache Airflow fundamentals.

References: Apache Airflow documentation, Dags; Microsoft Learn, Pipeline execution and triggers.