Airflow Pool

An Airflow pool limits concurrency for tasks assigned to a named shared capacity. Each task consumes pool_slots while counted as occupying that pool; a task can require more than one slot.

With four slots and tasks requiring two each, at most two such tasks can occupy the pool together if nothing else consumes slots. Other scheduler or executor limits can reduce this further. Slot weights represent an allocation policy, not an automatic measurement of database connections or CPU.

Create the pool before deploying tasks that reference it, and include all relevant workloads. Traffic outside Airflow is not limited by it. A pool is not a requests-per-second limiter or a guarantee of fair service between historical and current work.

In Airflow 3.1, pools can choose whether deferred tasks count toward occupied slots. Match that choice to whether a deferred task still ties up the external resource. Inspect actual queueing and external load before treating a configured limit as verified protection.

If an eight-slot pool is shared by extracts that need two slots and small checks that need one, three extracts leave two slots for other eligible tasks. Four extracts leave none. A task requesting nine slots cannot fit at all until configuration changes. Investigate slot occupancy together with task requirements when work remains queued.

See Production Airflow: Idempotency, Retries, and Backfills for worked examples.

Further reading

• Apache Airflow 3.1.0: Pools