Data Freshness

Data freshness describes how up to date data is for its intended use. A freshness measure needs a reference time, a measurement point, and an acceptable delay. A loading timestamp, the time of a business event, and the completion time of a report answer different questions — the same distinction as event time and processing time, applied to a delivery rather than to a stream.

Read the timestamp before reading the status

At 10:00, the most recent loaded row has an ingestion timestamp of 09:55. The age of that latest arrival is five minutes. But if it describes an event from yesterday, this does not show that today's activity is available.

Conversely, a quiet store may have no new sales for an hour even though its connection works. A timestamp measuring the newest event would grow older without a broken pipeline. The expected activity pattern matters when choosing an alert.

Freshness measurements observe different things. The age of the newest record compares max(loaded_at) with now; dbt’s field-based source freshness uses this kind of comparison against warning and error thresholds. The time since the last successful run observes pipeline activity and can remain healthy on empty input. Expected-partition checks test whether particular slices arrived. Scope matters as much as the metric: a global maximum can hide a stopped source behind another source’s fresh rows, while a maximum tracked for each expected source can expose that source’s staleness. A partition that exists can still lack rows. Freshness and completeness therefore need separate expectations, including which sources or slices should exist and what counts as complete delivery.

Which timestamp the first form reads decides what it can detect, and three different clocks get confused here. A load timestamp stored on each row measures when data last arrived: max(loaded_at) stops advancing as soon as new rows stop appearing, whether the pipeline is running or not, which is exactly why it catches a quiet source. A run heartbeat — a time the pipeline writes every execution, whether or not it loaded anything — measures the pipeline, and is the thing that cannot go stale while the pipeline runs. Those two are easy to conflate because both are written by the pipeline; they fail in opposite directions. A business event timestamp taken from the source measures the data as the source dated it, and grows old when the source stops or when events are delayed upstream of you.

Tools differ in which they can use, and the same tool differs by configuration and version. In dbt, providing loaded_at_field makes freshness a select query against that column or expression; a loaded_at_query may be given instead from v1.10 (setting both is a parse error); and since v1.7, leaving the field out makes dbt use warehouse metadata tables where the adapter supports it — for Snowflake, the LAST_ALTERED column — which records when the table was last touched rather than what arrived in it. So read your own configuration before you read the result: the three clocks above answer different questions, and a green check tells you only about the one it consulted.

Derive freshness thresholds from the deadline they protect. A report due at 08:00 with a 40-minute build needs its inputs by 07:20 if nothing else consumes time. A 07:00 threshold adds a 20-minute safety margin for variation or recovery. Stated for consumers, this becomes an objective with an explicit timing assumption. Passing it answers the defined freshness question, not every data-quality question.

A freshness target has an operating cost. More frequent full rebuilds can increase compute use; incremental processing can change that relationship and introduce its own state and recovery requirements. Compare the cost and reliability of concrete designs against the decision the consumer needs to make. A consumer who reads once each morning may still need changes from the preceding hour. Reading frequency alone does not determine whether fresher data has value.

Fresh does not mean complete

Imagine ten stores sending sales. Nine stop sending, while one delivers a record every minute. The latest timestamp across the whole table still looks recent. Check the expected stores or partitions separately, and compare expected arrivals when possible.

A freshness objective might require each active store's updates to become queryable within 15 minutes. This illustrative objective names both the scope and the delivery boundary. Accuracy and completeness still need their own checks — data quality assertions on the output, and reconciliation against the source for what never arrived. Late records that land after a window has closed are a third question again, covered by late data and lookback.

References

• dbt, Source freshness
• dbt, freshness configuration reference
• dbt, Upgrading to v1.7: behavior changes (which path is used, and from which version)