Data Incident

A data incident is a situation in which incorrect, incomplete, or missing data has reached consumers, and the response is closer to incident handling than to fixing a bug. The difference is that the damage can keep growing while you work: as long as the affected data is still being published or consumed, downstream runs can copy the problem further and reports can inform decisions made on the wrong number.

The order that works

1. Contain the impact and warn consumers. Withhold publication, quarantine affected output, or pause selected dependent jobs according to how the error can spread. Tell consumers which figures are suspected and whether to stop using them, including downloaded files and open reports. Do not wait for a complete diagnosis or stop every dependent job indiscriminately. Assign someone to update the warning as evidence changes; Google SRE describes stakeholder communication as an ongoing incident role.
2. Establish which periods, keys, downstream tables, and reports may be affected. Lineage narrows the search; query logs provide account-level access evidence and consumer candidates. A shared BI account or connection pool can represent several reports and people, while cached or exported results can be read without a new query. Combine query tags, BI metadata, application records, and owner confirmation when deciding whom to notify. Record gaps in retention and coverage rather than treating the log as a complete consumer list.
3. Correct and reprocess. Fix the cause, then apply the agreed correction to the affected range. When publishing a complete replacement snapshot, pin the input, definition, and scope; prevent an older run from overwriting a newer correction; and publish the validated result across the intended boundary without exposing a partial replacement. A correction-event design may instead append identified adjustments under a contract that prevents duplicate effects. Replacement alone does not make retries harmless: repeating the same logical correction must not apply its effect twice. Rebuilding a historical range is a backfill, with its own verification.
4. Confirm the correction and recovery. Follow up on the early warnings with the verified scope, revised figures, any remaining uncertainty, and when consumers can resume use. Where published figures changed, that is a restatement to announce rather than a quiet fix. A consumer who learns from a colleague that last week’s figure moved stops trusting every figure.

These activities overlap. Continue consumer updates while establishing the scope and repairing the data; the final correction notice closes that communication loop rather than starting it.

Who decides what

The roles in data owner, steward, and custodian map onto the response, and separating them prevents the most common failure — an engineer silently choosing what the right answer should be.

• The owner decides whether to withhold a figure, publish it with a caveat, or let it stand while the repair runs. This is a business judgment about which error is worse, and it belongs with whoever answers for the consequences.
• The steward establishes what happened, which records are affected, and what the corrected definition or value should be.
• The custodian runs the pause, the repair, and the reprocessing, and confirms afterward that the destination holds what it should.

Where one person holds all three roles, the decisions still have to be made in that order. The failure is not having one person; it is going straight to the repair and discovering afterward that nobody agreed what the correct answer was.

Evidence makes the second step possible

Data quarantine preserves rejected or suspect records with their reasons, making them useful evidence. Logging only a rejection count loses detail, though retained inputs may still allow reconstruction. Errors discovered after passing checks may never have entered quarantine. Use retained source snapshots, published versions, transformation history, lineage, and reconciliation to reconstruct the affected scope, and record any gaps that keep that scope uncertain.

Verification at the end deserves the same seriousness. A successful rerun proves execution completed. Reconciliation checks agreement with a specified source, time, and scope; it can pass while faithfully copying a wrong source value. If the incident concerns meaning or business facts, also validate the approved definition against expected results and appropriate independent evidence. Choose those checks for the failure being repaired rather than requiring a manual review of every incident. The principle is the one Google’s reliability engineering writing applies to backups — what people actually want is restores, and you only know you can recover if you actually do it.

Afterward

Three questions are worth asking once the incident is closed, and none of them is about who made the mistake. How long did it take to notice, and what would have noticed it sooner — a check, or an observability signal on volume or distribution? Could the problem have been caught before publication, by making the relevant check blocking rather than warning? And who found out from us versus from their own confusion, which is the measure of whether the notification step worked.

How incidents relate to quality checks, thresholds, and what a contract can promise is worked through in Passing Tests, Wrong Numbers.

References

• Google SRE Book, Managing Incidents
• Google SRE Book, Data Integrity
• GOV.UK, Data ownership model
• Snowflake — QUERY_HISTORY view