Data Pipeline

A data pipeline is a connected sequence of steps that reads data, moves or transforms it, and delivers an output. It can branch into several paths. A pipeline does not have to change every record: copying data to a destination can also be a pipeline.

Follow one order through the steps

Suppose an order system records a sale for \$40. A pipeline copies the record, checks that the order identifier is present, converts its timestamp to the reporting calendar, and includes the sale in a daily total. The input record, the rules applied to it, and the published total are different parts of the same process.

If the copy succeeds but the timestamp conversion assigns the wrong business date, the pipeline has run without producing the intended answer. Useful checks therefore examine the output as well as whether each step finished.

Pipeline, platform, and product

The pipeline is the processing path. The data platform supplies shared capabilities such as storage, execution, permissions, and monitoring. The data product is what a defined group of users can rely on, such as a documented daily sales table with an owner and a refresh commitment. Several pipelines may support one product.

ETL means extract, transform, load; ELT moves the transformation after loading. Both describe particular arrangements of pipeline steps. Neither term covers every possible pipeline.

Further reading

• AWS: Data pipelines