Data Incident

A data incident is a situation in which incorrect, incomplete, or missing data has reached consumers, and the response is closer to incident handling than to fixing a bug. The difference is that the damage keeps growing while you work: every downstream run that executes during the investigation copies the problem further, and every report opened is a decision someone may now make on the wrong number.

The order that works

1. Stop the spread. Pause the jobs that consume the affected output before changing anything. This is counterintuitive under pressure — the instinct is to fix first — but each further run adds to the set of things that will need correcting, and a pause is reversible while a propagated error is not.
2. Establish the extent. Which periods, which keys, which downstream tables and reports. A repair that cannot state its scope cannot be verified afterwards, and “we think it started Tuesday” is not a scope. Lineage narrows the search; query logs show who actually read the affected tables, including consumers no catalog knows about.
3. Correct and reprocess. Fix the cause, then recompute the affected range — replacing the affected output rather than appending a correction, so that a repeated run is harmless. Rebuilding a historical range is a backfill, with its own verification.
4. Tell the people who received the wrong numbers. Where published figures changed, that is a restatement to announce rather than a quiet fix. A consumer who learns from a colleague that last week’s figure moved stops trusting every figure.

Steps two and four are the ones that get skipped under time pressure, and they are the ones that determine whether the incident ends or merely stops being visible.

Who decides what

The roles in data owner, steward, and custodian map onto the response, and separating them prevents the most common failure — an engineer silently choosing what the right answer should be.

• The owner decides whether to withhold a figure, publish it with a caveat, or let it stand while the repair runs. This is a business judgment about which error is worse, and it belongs with whoever answers for the consequences.
• The steward establishes what happened, which records are affected, and what the corrected definition or value should be.
• The custodian runs the pause, the repair, and the reprocessing, and confirms afterwards that the destination holds what it should.

Where one person holds all three roles, the decisions still have to be made in that order. The failure is not having one person; it is going straight to the repair and discovering afterwards that nobody agreed what the correct answer was.

Evidence makes the second step possible

Establishing extent is only feasible if the rejected or suspect records still exist with their reasons attached, which is what data quarantine preserves. A pipeline that logs “1,204 rows rejected” and discards them has recorded the size of the problem and destroyed the ability to describe it.

Verification at the end deserves the same seriousness. A rerun that completes is not evidence that the data is now right; a reconciliation against the source is. The principle is the one Google’s reliability engineering writing applies to backups — what people actually want is restores, and you only know you can recover if you actually do it.

Afterwards

Three questions are worth asking once the incident is closed, and none of them is about who made the mistake. How long did it take to notice, and what would have noticed it sooner — a check, or a observability signal on volume or distribution? Could the problem have been caught before publication, by making the relevant check blocking rather than warning? And who found out from us versus from their own confusion, which is the measure of whether the notification step worked.

How incidents relate to quality checks, thresholds, and what a contract can promise is worked through in Passing Tests, Wrong Numbers.

References: Google SRE Book, Data Integrity; GOV.UK, Data ownership model.