Data SLIs, SLOs, and Incident Response

In this illustrative scenario, a finance team opens a revenue dashboard at 08:05 and sees ordinary-looking numbers. The pipeline that feeds it ran on schedule overnight, every task finished green, and no alert fired. Six hours earlier, one of five ingestion sources had silently stopped delivering, so the dashboard understates revenue by roughly a fifth. Nothing in the platform was broken in the way monitoring usually defines broken: no job failed, no query errored, and the table is only slightly out of date by its newest row. Someone eventually notices because a number looks low. The failure was detected by a consumer and not by the platform.

This article is about closing that gap. It defines what “working” means for a dataset in terms that can be measured, sets targets on those measurements, alerts on the consumption of the allowed failure, assigns someone to respond, and describes what to do, and what to tell consumers, when the data is wrong anyway. The structure follows the planned scope: freshness, completeness, and correctness as the three indicators; alerting; ownership; silent failures; and repair with consumer communication.

It assumes the previous article, Access, Retention, and Erasure Across Data Copies, only in one respect. That article argued that governance rules (who may read a copy, how long it lives, whether it was erased) cannot be asserted and must be evidenced. The same applies to reliability: a pipeline that reports success is making a claim, and this article is about what evidence makes the claim believable. An erasure job that silently stops is a silent failure of exactly the kind discussed here, and “erasure requests completed within the response period” is a perfectly good indicator to put under the same machinery.

Why a green job is not a reliable dataset

Service reliability practice starts from a simple move: define what users experience, measure it, and set a target. For a request-serving system the measurement is typically availability or latency. The Google SRE book observes that big-data systems such as pipelines tend to care about throughput and end-to-end latency (how long data takes to progress from ingestion to completion), and that all systems should care about correctness: whether the right data was retrieved and the right analysis done. It adds that correctness is often a property of the data in the system and not of the infrastructure, so it is usually not something infrastructure engineers can meet alone. For data platforms, that last point is the center of the problem. The people who run the job and the people who define a correct number are not always the same people.

A job-success signal describes execution. It answers whether the process started, ran, and exited without an error. It does not say what the process read, how much it wrote, or whether the result means what consumers think it means. A job can finish in twelve minutes after reading an empty file, or complete a join that multiplies rows. A scheduler that is paused produces no failures at all, because nothing ran. The distinction is the one the data observability entry draws between execution health (job outcomes, durations, errors) and output quality (record counts, freshness, content checks). Both are needed, but only the second describes what a consumer receives.

Data service-level thinking therefore needs indicators on the output. The Site Reliability Workbook, in its chapter on data processing pipelines, lists the common shapes of pipeline objectives. For freshness: some percentage of data processed within a time, the oldest data being no older than a time, or the pipeline job having completed successfully within a time. For correctness: golden or test data whose expected output is known, or a backward-looking target such as a maximum fraction of incorrect records per quarter. The sections below take these shapes and make them concrete.

Indicators: what to measure and how to define a good event

A service level indicator is a carefully defined quantitative measure of some aspect of the level of service provided. The workbook recommends expressing it as a ratio of good events to valid events. Two decisions are forced by that form, and both are where data teams usually need to be more explicit than service teams. What counts as a valid event: is a table that is legitimately idle on Sundays a failure, or excluded from the denominator? And what counts as good: arrived, arrived on time, arrived and complete, arrived and reconciled?

The workbook also separates an indicator’s specification, the outcome that matters to a user, from its implementation, the specification plus a way to measure it. For data, a specification such as “the orders table reflects every store’s sales within 30 minutes” can be implemented in several ways that see different things. The table below lists the three indicators in the scope of this article, an example specification and implementation for each, and what the implementation cannot see.

Indicator	Example specification (illustrative)	Example implementation	What it cannot see
Freshness	Each active store’s sales are queryable within 30 minutes of the sale	Per store, minutes since the newest loaded row, sampled once a minute; good if at most 30	A source that is legitimately quiet looks stale; a load that carries old events with a new load time looks fresh
Completeness	Every expected store-and-period slice is present once the delivery deadline has passed	Count of expected slices with at least one row, from an inventory that does not depend on arrivals	A slice that exists but is missing rows; records omitted from both the producer’s manifest and the extract
Correctness	Daily revenue per store equals what the payment system settled, within an agreed tolerance	Reconciliation of totals and keys against an independent reference, plus rule checks on values	Errors shared with the reference; meaning changes that keep totals consistent

Freshness: which clock you read decides what you can detect

The age of data depends on which timestamp is read, and three different clocks are easy to confuse. A load timestamp stored on each row records when data last arrived. Its maximum stops advancing as soon as new rows stop appearing, whether the pipeline is running or not, which is why it detects a source that has gone quiet. A run heartbeat, a time the pipeline writes on every execution whether or not it loaded anything, describes the pipeline and cannot go stale while the pipeline keeps running, so it cannot detect an empty load. A business event timestamp taken from the source measures data as the source dated it and grows old either when the source stops or when events are delayed upstream. Each answers a different question, and a green freshness check tells you only about the clock it consulted. Data freshness covers the same distinction in more detail.

Tooling makes this concrete. dbt’s source freshness configuration declares two thresholds, warn_after and error_after, and a loaded_at_field, the column or expression dbt queries to find how recently a source was loaded; its documentation presents these checks as an input to the service-level agreements a team defines. A source freshness check is thus a query of the form maximum of the load column compared with now, and it is only as informative as the column it reads. The dbt documentation also recommends running freshness jobs at least twice as often as the tightest freshness target they protect. A one-hour target checked once a day is not measuring the target.

Thresholds come from the deadline they protect. If a report is due at 08:00 and the build takes 40 minutes, its inputs must land by 07:20 at the latest. A 07:00 input threshold leaves a 20-minute margin for variation or recovery. This is arithmetic on an assumption (the build takes 40 minutes) and should be revisited when the build time changes. Notice also that the deadline is a calendar rule for a particular day’s delivery while the freshness threshold is the maximum age of the newest row when the check runs. They are related but not identical, and conflating them leads to thresholds that either fire constantly or never.

Completeness: define the expected inventory independently of what arrived

Completeness asks whether everything that should have arrived did. A check that only inspects arrived rows cannot fail on rows that never came, because an absent row cannot be selected. So completeness needs an expectation that exists before and independently of arrivals: a list of expected sources, a calendar of expected partitions, or a declared count from the producer. The delivery completeness entry adds the timing rule. A file due at 06:00 UTC the next day is not late at 08:00 on the day it describes, and “missing”, “not yet due”, “quarantined”, and “accepted” are different states. Counting every absent file as a failure produces false alarms during normal operation and teaches people to ignore the check.

The query below runs against a small synthetic table in DuckDB (version 1.5.6, local, in memory). The table holds one row per ten-minute slot for five stores over ten hours, and store 4 silently stops after 04:00. All data is invented for this article. The four queries are four different checks over the same table, and the output shows what each one sees.

import duckdb
con = duckdb.connect()
con.execute("""
CREATE TABLE sales AS
SELECT s AS store_id,
       TIMESTAMP '2024-11-04 00:00:00' + INTERVAL (m) MINUTE AS loaded_at,
       1 AS n
FROM range(1, 6) t(s), range(0, 600, 10) u(m)          -- 5 stores, one row per 10 minutes for 10 hours
WHERE NOT (s = 4 AND m >= 240)                           -- store 4 silently stops after 04:00
""")
con.execute("CREATE TABLE expected_stores AS SELECT * FROM range(1, 6) t(store_id)")
con.execute("CREATE TABLE manifest AS SELECT 300 AS rows_sent")       # producer declares 300 rows (5 stores x 60 ten-minute slots)

print("1. global freshness: age of newest row at 10:00")
print(con.execute("""SELECT date_diff('minute', max(loaded_at), TIMESTAMP '2024-11-04 10:00:00') AS age_min FROM sales""").fetchall())

print("2. per-source freshness at 10:00 (minutes)")
print(con.execute("""SELECT store_id, date_diff('minute', max(loaded_at), TIMESTAMP '2024-11-04 10:00:00') AS age_min
                     FROM sales GROUP BY 1 ORDER BY 1""").fetchall())

print("3. expected sources with no rows in the last hour")
print(con.execute("""SELECT e.store_id FROM expected_stores e
                     LEFT JOIN (SELECT DISTINCT store_id FROM sales WHERE loaded_at >= TIMESTAMP '2024-11-04 09:00:00') s USING (store_id)
                     WHERE s.store_id IS NULL""").fetchall())

print("4. rows loaded vs rows the producer declared")
print(con.execute("""SELECT count(*) AS loaded, (SELECT rows_sent FROM manifest) AS declared FROM sales""").fetchall())

1. global freshness: age of newest row at 10:00
[(10,)]
2. per-source freshness at 10:00 (minutes)
[(1, 10), (2, 10), (3, 10), (4, 370), (5, 10)]
3. expected sources with no rows in the last hour
[(4,)]
4. rows loaded vs rows the producer declared
[(264, 300)]

Reading the output. The first check, global freshness, reports that the newest row is 10 minutes old, which passes any reasonable threshold. The table as a whole is fresh because four of five stores keep sending data. The second check groups by store and shows the problem: store 4 is 370 minutes stale while the others are at 10. The third check is a completeness check against an expected inventory, and it names store 4 directly, with no threshold at all. The fourth reconciles the loaded row count against what the producer declared. 264 of 300 expected rows arrived, so 36 rows are missing, which equals six hours of store 4’s ten-minute slots. Each check is cheap, and the first is the one most dashboards implement. The lesson is not that global freshness is wrong but that scope is part of the definition: a global maximum can hide a stopped source behind another source’s fresh rows.

A caveat applies to the fourth check and to any manifest-based comparison. The manifest is the producer’s own statement. Agreement with it establishes that what was declared arrived, not that every real business event was declared. Stronger assurance needs an independent inventory, such as counts from the originating system itself, and the reconciliation entry discusses how the choice of reference sets the strength of the claim.

Correctness: the hard one

Correctness is the indicator that is hardest to measure because it requires a notion of the right answer. The SRE workbook offers two approaches, and both are general. The first is to generate expected output: where no predefined correct output exists, create it, for example by running test accounts through the production system so that the expected result is known, then comparing expected and actual output and alerting on discrepancies. The second is backward-looking: set a target such as no more than a stated fraction of records being incorrect per quarter, or limit the number of hours or days that known-bad data was served. The workbook warns that the notion of correctness varies by application, and the billing example it uses (charging customers too much or too little) shows why a data team cannot define it alone.

In practice a correctness indicator is assembled from several partial checks, each with a stated limit. Reconciliation of totals and keys against an independent reference detects dropped, duplicated, or altered records. Equal counts can hide different records, and equal totals can hide offsetting errors, so reconciliation compares keys and values and not only aggregates. Rule checks (the data quality dimensions of validity, uniqueness, consistency) detect forbidden values, and they pass on a value that is permitted but wrong, such as a valid status assigned to the wrong order. Contract checks at the boundary, as in a data contract, catch violations of declared units, categories, and keys. None of these proves that a number reflects reality. Agreement with a source proves only agreement.

One more property of the workbook’s treatment is worth carrying over: measure correctness end to end and not only per stage. Each stage can report that all is well while one stage introduces a field that a downstream stage was expected to process but does not. A check on the final published output catches what the sum of stage checks misses.

Setting the objective

An indicator becomes useful when a target is attached. An SLO is a target value, or range of values, for a service level measured by an SLI, and it only means something with a stated window. The specification below is an illustrative shape, not the syntax of any particular tool.

slo: orders_table_freshness
owner: commerce-data-product         # a role with authority to decide, not a person
consumers: [finance-dashboard, daily-revenue-report]
sli:
  good_event: "minute at which the newest row of every active store is at most 30 minutes old"
  valid_event: "every minute of the window, for stores flagged active at that minute"
  measured_at: "warehouse table, using the load timestamp column"
  no_data_policy: "a store with no rows ever is reported as unknown, not as good"
window: rolling 30 days
target: 99%                          # 432 bad minutes allowed in 43,200
exclusions: ["announced source maintenance windows, recorded in the exclusion calendar"]

The four fields that matter are the ones the SLO entry names: what counts as good, what counts as valid, where it is measured, and over what window. The no_data_policy line addresses a failure particular to ratio indicators: with no eligible events, the ratio is undefined, and reporting 100 percent would turn silence into success. The exclusions line is worth keeping short. Every exclusion lowers the number’s honesty, so each should be recorded with its reason and reviewed.

Choosing the target follows a few rules from the SRE material. Start from what consumers notice, not what is easy to measure. Do not copy current performance, because a target set at today’s best result commits the team to heroic effort to maintain it. Do not aim for 100 percent: the budget is the allowance for failure that makes change possible, and an unreachable target means every incident is a breach. And keep the number of objectives small, because a dashboard of targets nobody acts on adds noise. For a data pipeline the natural source of targets is the consumers’ deadlines, as in the report example above.

The arithmetic of the error budget is simple and worth doing explicitly. A 99 percent target over 30 days of one-minute samples allows 1 percent of 43,200 minutes, which is 432 bad minutes. A 44-minute stall consumes about 10 percent of that. A four-hour outage in which every minute is bad consumes about 56 percent in a single event. Those percentages, not the minutes, are what make incidents comparable and what the alerting rules below use.

An SLA is a different object. It attaches agreed consequences, such as a credit, an escalation, or a priority change, to a commitment, and engineers typically contribute a precise indicator while the business owns the terms. Internal data consumers can have agreements too, but a team’s own response to missing its internal SLO is not automatically an SLA. A tighter internal SLO than the commitment leaves operating margin, as long as both use the same definitions. If they use different windows or exclusions, they must be calculated separately.

The budget needs a policy to have effect. The error budget entry’s central point is that a calculation without an agreed response changes nothing. For a data pipeline the response options are different from a web service’s, so be explicit. Examples of actions a policy can bind: pause non-urgent schema changes and backfills on the affected pipeline until the budget recovers; assign the team to reliability work ahead of new consumer requests; require an additional review for changes to the ingestion layer. The policy should name the owner who can enforce it, the exceptions (urgent security and recovery changes), and the conditions for resuming normal work. An advisory budget with no one empowered to act is a report.

Alerting: page on consumption of the budget, not on every breach

The simplest alert is a threshold on the indicator: page when freshness is worse than 30 minutes. For a single dataset it looks sensible and it fails in two ways. It fires on every short excursion, including ones that consume a trivial share of the budget, and it cannot distinguish a one-minute lapse from a six-hour outage. The workbook shows the cost of this precision problem: with a 99.9 percent monthly objective, a 0.1 percent error rate sustained for ten minutes consumes only about 0.02 percent of the monthly allowance, yet a rule that notifies on it can produce up to 144 notifications a day while the objective is still met. The alert fatigue that follows is a design problem in the rule, not a weakness of the people receiving it.

The alternative is to alert on how quickly the budget is being consumed. Burn rate is the observed error ratio divided by the error ratio the SLO allows. For a 99 percent target the allowed ratio is 0.01, so a window in which 1 percent of minutes are bad has burn rate 1, and one in which 14.4 percent are bad has burn rate 14.4. A rate of 1 spends exactly the budget over the full window. The thresholds in the workbook come from choosing a fraction of the budget that is worth waking someone for and the window over which it is spent.

Notification	Budget fraction consumed	Long window	Short window	Burn rate threshold	Derivation
Page	2%	1 hour	5 minutes	14.4	0.02 / (1/720), since one hour is 1/720 of 30 days
Page	5%	6 hours	30 minutes	6	0.05 / (6/720)
Ticket	10%	3 days	6 hours	1	0.10 / (72/720)

Each rule fires only when both its windows are above the threshold. The long window establishes that a significant share of the budget has actually been spent. The short window makes the alert reset soon after recovery, because otherwise a one-hour window keeps a rule firing for up to an hour after the problem ends. The workbook suggests a short window of about one-twelfth of the long one, and it presents the percentages above as reasonable starting points that depend on the service. Pages go to a person immediately, tickets to a queue, which is how the rules express that a fast burn and a slow burn deserve different responses.

The simulation below applies these three rules to a synthetic month. A batch loads every 15 minutes, freshness age is the minutes since the newest batch, and a minute is good if the age is at most 30. Three invented incidents are injected: a 50-minute stall on day 3, a four-hour stall on day 12, and a slow degradation on days 20 to 23 in which, every six hours, one load arrives 23 minutes late. It was executed locally with Python 3.12, using only the standard library.

# Synthetic freshness SLI over 30 days, one sample per minute.
# A batch loads every 15 minutes. Freshness age = minutes since the newest loaded batch.
# A minute is "good" if age <= 30. All incidents are invented for illustration.
DAYS = 30
N = DAYS * 24 * 60
SLO = 0.99                      # 99% of minutes good over 30 days
BUDGET = (1 - SLO) * N          # allowed bad minutes

def stalled(t):
    d, m = divmod(t, 1440)
    if d == 3 and 600 <= m < 650:   return True    # I1: 50-minute stall
    if d == 12 and 120 <= m < 360:  return True    # I2: 4-hour stall
    return False
def slow_burn_day(t): return 20 <= t // 1440 < 24
def skipped(t):                      # I3: on days 20-23, at every 6-hour mark the load is moved from t to t+23
    if not slow_burn_day(t): return False
    k = t % 360
    return k in (0, 15)
def extra_load(t):
    return slow_burn_day(t) and t % 360 == 23

last_load = 0
age = []
for t in range(N):
    if (t % 15 == 0 and not stalled(t) and not skipped(t)) or extra_load(t):
        last_load = t
    age.append(t - last_load)
bad = [1 if a > 30 else 0 for a in age]

print("bad minutes:", sum(bad), "of", N, "| budget:", BUDGET, "| consumed: %.1f%%" % (100 * sum(bad) / BUDGET))
print("job success rate in the same period: 100% (every scheduled run that started, succeeded)")

def window_rate(t, w):
    lo = max(0, t - w + 1)
    return sum(bad[lo:t + 1]) / (t + 1 - lo)
RULES = [("page 14.4x  1h & 5m", 60, 5, 14.4), ("page 6x     6h & 30m", 360, 30, 6.0), ("ticket 1x  3d & 6h", 4320, 360, 1.0)]
for name, lw, sw, thr in RULES:
    fired = [t for t in range(N) if window_rate(t, lw) > thr * (1 - SLO) and window_rate(t, sw) > thr * (1 - SLO)]
    if fired:
        # group into episodes
        eps, start, prev = [], fired[0], fired[0]
        for t in fired[1:]:
            if t != prev + 1: eps.append((start, prev)); start = t
            prev = t
        eps.append((start, prev))
        print(name, "->", [f"day {s//1440} {s%1440//60:02d}:{s%60:02d} for {e-s+1} min" for s, e in eps])
    else:
        print(name, "-> never fired")

bad minutes: 380 of 43200 | budget: 432.0000000000004 | consumed: 88.0%
job success rate in the same period: 100% (every scheduled run that started, succeeded)
page 14.4x  1h & 5m -> ['day 3 10:24 for 40 min', 'day 12 02:24 for 220 min']
page 6x     6h & 30m -> ['day 3 10:37 for 51 min', 'day 12 02:37 for 231 min']
ticket 1x  3d & 6h -> ['day 3 10:59 for 357 min', 'day 12 02:59 for 537 min', 'day 21 12:17 for 3602 min']

Reading the output. The month consumed 88 percent of the budget: 44 bad minutes from the first stall (about 10 percent), 224 from the second (about 52 percent), and 112 from the slow degradation (about 26 percent). Throughout, the job success rate is 100 percent by construction, since every run that started succeeded, so an indicator based on job success reports a perfect month. The page rules fire for both stalls. The 14.4 rule fires at 10:24 on day 3, eight minutes into the bad period, which is when the nine-minute bad count crosses 14.4 percent of the one-hour window. It stops firing once the short window clears, which is the reset behavior the second window provides.

The slow degradation is the instructive case. Each event leaves only 7 bad minutes, which stays below the page thresholds in both the one-hour window (7 of 60 is 11.7 percent, below 14.4) and the six-hour window (7 of 360 is 1.9 percent, below 6). No page fires. The ticket rule does fire, on day 21 at 12:17, after about a day and a half, once the three-day window average crosses 1 percent. Without the slow rule, this pattern would have consumed the 26 percent of budget silently, and the rule that did catch it was the one that sends a ticket, not a page, which matches its urgency: at that rate a full budget would last about 15 days.

Three limits of burn-rate alerting matter for data in particular. First, sampling granularity: the simulation samples a continuously available age metric every minute. A daily table that is either on time or not produces one sample per day, a single bad day is a 100 percent error ratio over any shorter window, and the one-hour rule is meaningless. For batch datasets, alert on deadlines instead (expected completion time versus now, with the not-yet-due state handled) and use burn-rate arithmetic for the monthly or quarterly view. Second, small counts: with few events per window, a single bad event produces a huge burn rate, and the response should depend on its impact and not only the ratio. Third, overlap: a severe event crosses all three rules and should produce one grouped incident and not three notifications, so grouping and suppression need explicit configuration and tests.

Alerting on the indicator is not the whole of detection. Dataset-level checks for completeness and correctness need their own paths: a failed expected-inventory check at the delivery deadline is a page for the owner of that delivery, and a reconciliation difference above tolerance is a ticket or a page depending on the consumers affected. What the burn-rate framing contributes is a common currency, since incidents of different kinds (late, missing, or wrong data) can all be expressed as budget consumed, provided each is measured on the same good-event basis.

Ownership: who decides, who fixes, who tells the consumers

An alert that reaches no one who can act is not an alert. For data the ownership question is more tangled than for services, because the people who operate a pipeline, the people who define what its output means, and the people who accept the risk of a wrong number are frequently different. The three-role distinction in data owner, steward, and custodian fits the response well. The owner is accountable for decisions within the data scope, such as whether to withhold a figure or publish it with a caveat. The steward maintains meaning and coordinates routine quality work: establishing what happened, which records are affected, and what the corrected value or definition should be. The custodian operates the storage and controls: pausing jobs, running repairs, confirming the destination afterward. One person can hold several roles. The failure to avoid is going straight to a repair and discovering later that nobody agreed what the correct answer was.

A data product owner adds the consumer-facing duties: what the fields mean, what is promised and what deliberately is not, which request is built next, and when to deprecate. For reliability the practical consequence is that someone must own the SLO document itself, the list of consumers it protects, and the decision to renegotiate the target when it is persistently missed or never approached. Without named consumers there is nothing to measure satisfaction against.

Signal	Who is notified	Who decides what to do	Who acts
Freshness page (fast burn)	On-call for the pipeline	Pipeline owner or delegate: withhold, caveat, or let run	Custodian: restart, rerun, escalate to the upstream owner
Freshness ticket (slow burn)	Team queue	Owner prioritizes against the budget policy	Custodian or the team’s reliability work
Completeness check failed at deadline	Owner of that delivery (the source’s producer or the ingestion team)	Steward: is it missing, late, or legitimately empty?	Producer fixes delivery; custodian reloads
Reconciliation difference over tolerance	Steward and owner	Owner: publish, caveat, or withhold; steward defines the right value	Custodian reprocesses; steward verifies
Contract violation from an upstream source	Producer’s on-call	Producer’s owner	Producer fixes; consumers pause or quarantine

Cross-team ownership is where this breaks down most often. An ingestion pipeline depends on source systems owned by other teams, and an incident often has its cause there. The producer’s side needs its own checks that implement the contract, and the consumer’s side needs a way to detect violations. A contract document alone does not enforce itself. The workable pattern is for the producer to publish expectations (freshness, volume, schema, accepted values) and for the consuming pipeline to check them at the boundary, so that a violation is detected where data enters and attributed to the producer, rather than detected three tables downstream as a wrong dashboard.

Pager load is an ownership concern too. A rotation needs enough people and enough capacity for follow-up work. The on-call rotation entry reports the Google model’s measurement that an incident, including root-cause analysis, remediation, and postmortem follow-up, averages about six hours of work, which implies a ceiling on incidents per shift; the figure is a measurement from that organization and should be replaced by local data. For a small data team that rotates the pager among five people, a pipeline that produces daily false pages is not only an annoyance. It consumes the capacity needed to fix the causes.

Silent failures: when every signal says healthy

A silent failure is one where the data is wrong or missing while the usual signals report success. They are the central risk of data systems because the usual signals were designed for services. The table lists recurring ones, why the job stays green, and which detector sees each. It is a working taxonomy.

Failure	Why the job succeeds	What detects it	What still escapes
A source stops delivering	Empty input is valid input	Per-source freshness; expected-inventory check; volume against the source’s usual level	A source that is legitimately quiet at that hour
Scheduler paused or deleted	Nothing ran, so nothing failed	Freshness of the output; deadline alert for the expected run	A heartbeat metric alone, which also stops
Partial load	The run completed and loaded what arrived	Reconciliation against declared or independent counts	Records missing from both the extract and the manifest
Duplicated rows from a join	A join that multiplies rows is a valid join	Uniqueness check on the grain; reconciliation of totals	A duplicate that changes no total
Meaning change upstream	Types and names are unchanged	Contract checks on accepted values and units; distribution monitoring; reading release notes	A change that keeps the distribution the same, such as a redefinition with similar frequencies
Late data after the window closed	The window was processed as scheduled	Late-arrival counts; reconciliation after the lookback period	Data later than any lookback
Stale serving copy	The warehouse is correct; the cache or extract is old	Freshness measured at the consumer’s access point	A manually exported copy outside the graph
Wrong but consistent	Source and target agree	Independent reference; golden-data tests; consumer feedback	Errors shared by every reference available

The data observability entry organizes the monitoring for these into four signals: volume (rows or bytes per load, per partition, per source), freshness (per source, not overall), schema (columns and types as they arrive), and distribution (null rates, category proportions, value ranges). Volume is the cheapest and often the most informative. Freshness, as the DuckDB output above shows, needs to be per source to be useful. Distribution checks catch some meaning changes that type checks miss. The same entry makes two points that apply directly here. Assertions you can state in advance (an expected set, a count, a reconciliation total) are cheaper and fail with a clearer reason than an anomaly alert, so write those first. And anomaly detection compares today with how the dataset normally behaves, so it needs history and produces false alarms for legitimate changes such as a campaign or a new region; there must be a way to record that a change was intended, or people learn to ignore the alerts.

Some meaning changes cannot be detected from the data at all. If a status code keeps the same frequencies while its definition changes, nothing looks anomalous. Those are found by reading the change (a release note, a schema review, a question to the producing team), which is why the producer relationship is part of the control and not an alternative to monitoring.

Incident response: contain, scope, repair, confirm

When wrong data has reached consumers, the response has the shape of incident handling more than bug fixing, because damage grows while you work: each downstream run during the investigation copies the problem further, and each report someone opens is a decision that may rest on a bad number. A data incident is such a situation, and the order that works is a sequence of overlapping activities.

1. Contain and warn. Withhold publication, quarantine the affected output, or pause selected dependent jobs, according to how the error can spread. Tell consumers which figures are suspected and whether to stop using them, including downloaded files and open reports. Do not wait for a full diagnosis, and do not stop every dependent job indiscriminately. Someone should own updating the warning as evidence changes. An incident roles model (an incident commander coordinating, with communication and operations separated) helps on larger incidents; the incident commander entry describes the role.
2. Establish scope. Which periods, keys, downstream tables, and reports could be affected? Lineage narrows the search, and query logs provide account-level access evidence. A shared BI account can represent many people, and cached or exported results can be read without a new query, so combine logs with BI metadata, owner confirmation, and consumer interviews, and record the gaps.
3. Correct and reprocess. Fix the cause, then apply the agreed correction to the affected range. Rebuilding a historical range is a backfill, with its own verification. When publishing a replacement, pin the input, definition, and scope, prevent an older run from overwriting a newer correction, and publish the validated result across the intended boundary without exposing a partial replacement. Repeating the same logical correction must not apply its effect twice, which is the property to design the repair job around.
4. Confirm and close. Follow up on the early warning with the verified scope, revised figures, remaining uncertainty, and when consumers may resume. Where published figures changed, that is a restatement to announce and not a quiet fix. A consumer who learns from a colleague that last week’s figure moved stops trusting every figure.

Verification at the end deserves the same care as detection. A successful rerun proves that execution completed. Reconciliation proves agreement with a specified source, time, and scope, and can pass while faithfully copying a wrong source value. If the incident concerns meaning, validate the approved definition against expected results and independent evidence. This is the data analog of the point Google’s reliability writing makes about backups: what people want is restores, and you only know you can recover by actually recovering.

Telling consumers: content that supports a decision

Communication is part of the repair, not an afterthought, and the quality of a notice is whether a consumer can decide from it what to do. The workbook’s Spotify case study, a public account of an event-delivery system, illustrates why the content differs by failure type. It reports that when the data freshness SLO is broken, no customer action is expected, since customers wait for the data. When completeness or skew objectives are breached, customers might need to be involved because data quality is compromised, and the affected events need reprocessing, which in that account is done manually, with a recommendation that customers reprocess the delivered events afterward. The general point, which is an interpretation and not a claim about that system, is that a lateness incident asks consumers to wait while a wrongness incident asks them to act, so the notice has to say which one it is.

An illustrative notice template for a wrongness incident follows. Every field answers a question the consumer would otherwise have to ask.

Subject: [Data incident] Daily revenue understated for 2024-11-04 (store 4 missing)

Status:      Contained. Corrected figures expected by 14:00 UTC.
What is wrong: Revenue for 2024-11-04 excludes store 4 from 04:00 to 10:00.
              Reported total is understated by about one fifth for that period.
Affected:    finance-dashboard (daily view), daily-revenue-report, any export
             downloaded between 04:00 and 10:30 on 2024-11-04.
Not affected: Months before November; other stores; order-level tables.
What to do:  Do not use the 2024-11-04 revenue figure until the next update.
             Re-download exports after the correction notice.
Next update: 12:00 UTC, or sooner if the scope changes.
Owner:       commerce-data-product on-call

The template does three things that are easy to omit under pressure. It states what is affected and what is not, which stops consumers from discarding everything. It gives a time for the next update even if the news is that there is none. And it tells consumers about copies that the platform cannot reach (downloaded files), which connects to the previous article’s point that outbound copies leave the platform’s control. After the incident, send a closing notice with the verified scope and corrected values.

After the incident: the postmortem

A postmortem records impact, response, contributing causes, and follow-up work, and the blameless version examines actions and system conditions without indicting a person. The incentive is practical: if reporting an error invites punishment, people withhold information that an investigation needs. For a data incident the postmortem should answer three questions. How long did it take to notice, and what would have noticed sooner, a check or an observability signal? Could the problem have been caught before publication by making a check blocking rather than warning? And who found out from the platform versus from their own confusion, which measures whether the notification worked. The SRE material is blunt about action items: a postmortem without tracked, prioritized follow-up work changes nothing, so each user-affecting incident should produce at least one tracked action with an owner, an observable completion condition, and a verification method.

Measuring the reliability system itself

The measurement system needs its own measures, or it decays unnoticed. The table lists a small set. They answer whether the detection, response, and communication parts work and not only whether the data was reliable this month.

Measure	Definition	How to read it	Limit
Time to detect	From the start of the bad period to the first alert or human recognition	Compare with the delivery deadline and the consumers’ tolerance	The start of a silent failure is known only after the investigation
Detected by consumer	Share of incidents first reported by a consumer	The direct measure of whether the platform sees what consumers see	Small counts; a few incidents move the share a lot
Budget consumed per incident and per month	Bad minutes or bad slices divided by the budget	Compare incidents on one scale; watch for slow burns	Depends on the indicator definition
Page precision	Share of pages that required action	Low precision signals alert fatigue risk	Does not count missed incidents; pair with detection delay
Notice completeness	Incidents with a consumer notice that included scope, action, and next update	Whether the communication step works	Content checklist, not consumer comprehension
Open follow-up actions past due	Postmortem actions not completed by their date	Whether learning turns into change	A closed item still needs verification of its outcome

Failure modes and limits

• Indicators that measure the job. Success rate and duration describe execution. Put at least one indicator on the output a consumer reads.
• Global scope hiding a stopped part. Compute freshness and completeness per source or slice, with an expected inventory that does not depend on arrivals.
• Targets copied from current performance. A target at today’s best result converts every normal variation into an incident. Derive targets from consumer deadlines.
• Alerts with no owner or no action. Each rule needs a recipient who can act and a stated response. A rule that fires into an unattended channel provides no coverage.
• Burn-rate rules on coarse samples. A daily table cannot support a one-hour window. Use deadline alerts for batch datasets.
• Agreement mistaken for correctness. Reconciliation proves agreement with a reference. State the reference and its limits.
• Quiet fixes. A silent correction that changes published figures erodes trust more than the original error. Announce restatements.
• Postmortems that produce documents but not changes. Track the follow-up actions and verify their outcomes.

None of this guarantees that data is right. What it provides is a defined meaning for “working,” a measured answer to whether the platform met it, an alerting scheme that distinguishes fast from slow consumption of the allowance, named people who decide and act, and a communication practice that lets consumers act on the situation. The honest limits (meaning changes the data cannot reveal, references that share errors, copies that left the platform) should be stated in the SLO document, because consumers trust a stated limit more than an unqualified promise.

The next article, Disaster Recovery and Cost Engineering for Data Platforms, asks what happens when the failure is not wrong data but lost or corrupted storage, and what it costs to be ready for it.

References

• Google, Site Reliability Engineering: Service Level Objectives
• Google, The Site Reliability Workbook: Alerting on SLOs
• Google, The Site Reliability Workbook: Data Processing Pipelines (includes the Spotify event delivery case study)
• Google, Site Reliability Engineering: Postmortem Culture: Learning from Failure
• dbt documentation, freshness configuration
• dbt documentation, Source freshness