Sensor
A sensor is a task whose only job is to wait until a condition becomes true — a file exists, a partition is populated, an external system reports ready — and then let the rest of the workflow proceed. Airflow’s documentation describes sensors as operators “designed to do exactly one thing: wait for something to occur,” whether time-based, a file, or an external event; they succeed when the condition is met and fail on timeout so that the failure can raise an alert. Details follow Airflow documentation checked in September 2026.
Why wait rather than schedule later
A pipeline that starts at 07:00 because the input “usually arrives by 06:45” encodes a guess. On the night the upstream system is slow, the job reads nothing and reports success. A sensor replaces the guess with an observation: the downstream work begins when the data is actually there, and if it never arrives, the failure is explicit and points at the right cause.
What waiting costs
| Mode | How it waits | Suits |
|---|---|---|
| Poke (default) | The sensor “takes up a worker slot for its entire runtime” | Frequent checks, where low latency matters |
| Reschedule | Takes a worker slot only while checking, and sleeps between checks | Checks a minute or more apart |
| Deferrable | Hands the wait to a separate triggerer process, freeing the slot | Long waits, many concurrent sensors |
The failure this table prevents is a familiar one: a few dozen sensors in poke mode, each holding a worker slot for hours, and a cluster that cannot run the work it is waiting for. The checks themselves also cost something at the other end — polling a database or listing a bucket every few seconds is load on someone else’s system.
Settings that decide the behavior
poke_interval— seconds between checks. Frequent enough to be timely, sparse enough not to be a load test.timeout— the deadline. A sensor without one waits indefinitely, which turns a missing input into an outage that looks like patience rather than a failure anyone is paged for.soft_fail— marks the sensor skipped rather than failed on timeout, which propagates differently downstream. Useful for optional inputs, wrong for required ones.exponential_backoff— lengthens the gap between checks as waiting continues, reducing pressure on the system being polled.
When not to use one
Waiting is the fallback for when nothing tells you. If the producer can emit an event on arrival, an event trigger removes the polling entirely; if the orchestrator supports scheduling on dataset updates, declaring that dependency is clearer than a sensor plus a comment. Sensors remain the right answer for systems you do not control and cannot ask to notify you — which, in practice, is many of them. How this fits with triggers, dependencies, and failure handling is worked through in Running Pipelines in the Right Order.
References: Apache Airflow documentation, Sensors.
Discover more from Insightful Data Lab
Subscribe to get the latest posts sent to your email.