Data Incident
Bad data in production is closer to an incident than a bug. While the affected data keeps being published, copied, or read, the decisions and outputs needing correction can grow.
-
-
Cascading Failure
A cascading failure spreads because each response to overload adds load elsewhere — which is why the system can stay down after the trigger is removed.
-
Case: An Approved Agent for Transactional Work
A fictional company’s refund and reshipment agent, built and run locally: delegated credentials, approvals tied to one call, crash-safe steps, reconciliation with the ledger, and a policy-release incident drill.
-
Kill Switch
A kill switch is an operator-controlled setting that turns off a function at run time, without a code change, once each reader picks up the change. It helps in an incident only if it is read where the effect happens and fails in a chosen direction.
-
Incident Containment
Containment limits further harm during an incident, before anyone understands the cause. It is separate from removing the cause and from restoring normal service, and it carries its own cost.
-
Data SLIs, SLOs, and Incident Response
A green job does not mean a reliable dataset. Learn to define data indicators, set objectives, alert on budget burn, find silent failures, and tell consumers what to do.
-
Incident Response and Recovery Testing
Recovery is a claim until it has been exercised. This article covers incident roles, runbooks, containment, RTO and RPO, restore tests, and analysis.