Data Ingestion

Data ingestion is the process of bringing data from a source into a target environment where it can be stored or processed. It may read files, request pages from an API, or consume a continuous flow of records. The source system provides the records; ingestion performs the transfer.

Receiving a file is only one possible boundary

Imagine an API that returns at most 100 orders per response. There are 250 orders to retrieve. A successful first response delivers 100 orders, but ingestion is incomplete until the remaining pages have been handled. The last successful request time alone cannot prove that all 250 arrived.

A useful ingestion record identifies what was requested, what was received, and where processing can resume. For files, this might include a filename and expected row count. For a stream, it might include a position in the source log. The correct checkpoint depends on the interface.

Ingestion and transformation can overlap

An ingestion step may decode a message or validate required fields. More extensive business transformations, such as calculating net sales, may happen later. ETL and ELT place loading and transformation in different orders; ingestion is not a synonym for the entire pipeline.

In the API example, a retry could deliver the same page twice. Reliable ingestion therefore also needs a way to handle repeated records and to detect gaps. Getting data into storage is a boundary to verify, not proof that it is ready for business use.

Reference: AWS: Data pipelines.