Incident Containment

Incident containment is the set of actions that stop an incident from causing more harm while its cause is still being investigated. It answers the question “how do we keep this from getting worse right now,” and leaves two other questions to later steps: why it happened, and whether the system is healthy again. Google’s site reliability engineering workbook makes the same ordering explicit for outages, describing mitigation of the impact as the step before root-cause analysis, with rolling back a recent release as a typical mitigation applied before the cause is understood.

Three jobs in one incident

Incident handling is commonly split into containment, eradication, and recovery. The split matters because each job has a different finish line, and an action that satisfies one can be mistaken for completing another. The table uses a fictional support assistant whose new prompt dropped the rule that large refunds need a human approval.

Containment does not undo anything that already happened. Refunds issued before the tool was disabled stay issued, and an answer a customer has read stays read. It also leaves the defect in place, which is why a contained incident is still open.

A single action can serve more than one job. Repointing the assistant to the previous release bundle stops the new behavior, so it contains the incident, and it is a rollback. The defective prompt still exists in the new release, and eradication has not started. The job is defined by what the action is for in the incident, and the name of the action does not settle it.

Comparing containment actions

Two quantities separate containment actions. The first is exposure, the harm that still occurs before the action takes effect. If the bad behavior produces harmful events at a steady rate, exposure is that rate multiplied by the time from alert to full effect, which includes the time for a person to carry out the action and the time for the change to reach every component that acts. The second is the legitimate work the action blocks while it is in place. The script below uses assumed numbers: 0.8 wrong refunds per minute, and delays and blocked volumes chosen for illustration. It ran on Python 3.12.14 on macOS.

HARM_PER_MIN = 0.8         # wrong refunds the bad release causes per minute while active
# (action, minutes for a person to carry it out, seconds for it to reach every executor, legitimate actions blocked per hour)
ladder = [
    ("disable refund tool",          2, 10,   20),
    ("read-only mode (all writes)",  3, 10,   60),
    ("repoint alias to prior bundle",15, 300,  0),
    ("block all traffic",            4, 5,   600),
]
print(f"{'action':<32}{'seconds to effect':>18}{'wrong refunds let through':>27}{'legit actions blocked/hour':>28}")
for name, mins, prop, blocked in ladder:
    sec = mins * 60 + prop
    print(f"{name:<32}{sec:>18}{HARM_PER_MIN * sec / 60:>27.1f}{blocked:>28}")

action                           seconds to effect  wrong refunds let through  legit actions blocked/hour
disable refund tool                            130                        1.7                          20
read-only mode (all writes)                    190                        2.5                          60
repoint alias to prior bundle                 1200                       16.0                           0
block all traffic                              245                        3.3                         600

Disabling the refund tool takes effect in 130 seconds, lets about 1.7 wrong refunds through, and blocks 20 legitimate refunds per hour. Repointing to the previous bundle blocks no legitimate work, but a verified change plus a 300-second alias cache means 1,200 seconds and 16 wrong refunds. Blocking all traffic is fast and thorough and blocks the most. With these numbers the narrow, fast action goes first and the slower, precise one follows. After the rollback is verified the tool can be turned back on, because a containment action left in place keeps costing the blocked work.

What makes containment effective

• Speed and reach. The action must take effect in every component that can cause the harm, not only the one an operator changed. A kill switch that executors read at the moment of the effect reaches in-flight work that a check at request admission misses.
• Proportion. Use the least disruptive action that is enough, and record what it blocks.
• Independence. The control must not depend on the failing component, so an assistant cannot be the tool used to switch itself off.
• Evidence. Keep the faulty release, logs, and request records. Containment that discards them leaves later analysis with less to work from.
• Verification. Measure the harmful event rate after the action. A command that returned success does not show that every replica has stopped.

References

• Google, The Site Reliability Workbook: Incident Response