Airflow XCom

Airflow XCom is a mechanism for passing values between task instances. TaskFlow can automatically connect a task’s returned value to another task’s input. This does not mean the functions execute together in one Python process.

Use small metadata such as an accepted-row count or a versioned storage reference. Keep large datasets in suitable shared storage. XCom backend choices affect storage behavior, but changing the backend does not remove serialization, retention, access, or payload-size considerations.

A URI needs the intended immutable version, downstream access, and a retention policy that keeps it usable. A worker-local path may be inaccessible from another worker. Do not treat XCom as the authoritative data archive or place credentials in its values.

See Apache Airflow Fundamentals: DAGs, Tasks, and Scheduling for worked examples.

Reference: Apache Airflow 3.1.0 — XComs.