Incident Response and Recovery Testing

Consider a platform team whose release, at some point in the afternoon, begins writing wrong values to a payments table. Nothing alerts, because every request still returns success. Backups keep running every fifteen minutes, and each one faithfully copies the damage. A customer reports a discrepancy at the end of the day. Several things now decide how the next hours go: whether someone is clearly in charge, whether the first containing action stops the damage instead of changing the evidence, whether a recovery target exists and is realistic, whether a backup can actually be restored and which one is good, and whether the team will learn the right things afterward.

This article treats those as one subject, which is how a team gets from “something is wrong” to “we are back, and we know why.” It covers incident roles, runbooks, containment, recovery objectives, restore tests, disaster exercises, and incident analysis based on evidence.

The previous article in this series, Capacity, Load Shedding, and Resilience Testing, covered how a service behaves under overload, how to protect goodput with admission control, how to inject faults on purpose, and how to verify that a service has recovered and not merely lost its trigger. This article assumes you took those away. In particular it reuses the idea that recovery is a measured state with criteria and a time limit. It does not repeat overload controls. It addresses the incidents that need a human response and a recovery path, such as a bad change, lost or corrupted data, or the loss of a failure domain.

All examples are illustrative. The services, timelines, and numbers are invented for this article. The restore demonstration is a Python program written for it that runs locally on toy data with the standard library and SQLite; its output is reported as produced. The PostgreSQL commands are quoted as configuration sketches from the documentation and were not executed. The incident timeline later in the article is hypothetical, built on the demonstration’s scenario, and not a record of any real event.

Declaring an incident

An incident process begins with a decision that is often postponed: that this is an incident. The SRE book’s incident management chapter gives criteria for declaring one that avoid arguing about severity during the event. Declare an incident if any of the following holds: a second team’s involvement is required, the outage is visible to customers, or the problem is still unsolved after an hour of focused analysis. The SRE workbook’s incident response chapter reinforces the point with case studies in which declaring late prolonged resolution, and states that declaring early prevents miscommunication between teams and that managed incidents are resolved faster. Whether those results generalize to your organization is a question you can answer by measuring your own time to mitigate, but the argument for fixed criteria stands on its own: criteria agreed in advance remove a debate that no one has time for during the event.

Declaring does not mean escalating to the whole company. It means switching from ordinary work to a mode with named roles, a shared channel, and a written record. Many incidents are small and end within minutes, and the process scales down: one person holds every role and says so.

Roles

What the roles are

The SRE book describes a response structure adapted from the Incident Command System, built on a recursive separation of responsibilities: each person knows their role, and when the load exceeds what one person can handle, they delegate part of it. The incident commander entry covers the role in more detail. The four roles are:

Role	Responsibility	What goes wrong without it
Incident commander	Holds the high-level state of the incident, structures the response, assigns roles, and holds every role not yet delegated	Responders duplicate work or pull in different directions, and nobody notices when the plan has stalled
Operations lead	Applies the operational tools to mitigate; in the SRE book’s model, the only group changing the system during the incident	Several people change production at once and neither they nor anyone else can say which change did what
Communications lead	Issues updates to the responders, stakeholders, and users, and keeps the record accurate	The engineers answer questions and stop working, or stakeholders invent their own picture
Planning lead	Handles longer-term items such as filing bugs, arranging handoffs and food and rest, and tracking how the system diverged from baseline (emergency changes that must be undone)	Temporary changes are forgotten and become the next incident

The commander coordinates, communicates, and controls, which the workbook calls the three Cs: assign the work, keep participants and stakeholders informed, and maintain a shared view of what is happening.

Why the separation matters

The SRE book’s illustration is an unmanaged incident in which a responder focuses narrowly on a technical detail and loses the broader picture, communication leaves colleagues uncoordinated, and another engineer deploys a change without coordinating with anyone (“freelancing”), making the outage far worse. The lesson of the third failure is the reason for restricting changes to one role: when two people act on production independently, the system’s state is the product of actions that no one can reconstruct, and diagnosis becomes harder in proportion.

Choose the commander for the ability to coordinate and not for technical depth. The person who knows the failing system best is often more useful in the operations role, but expertise does not disqualify anyone from command; the constraint is attention. If debugging prevents the commander from tracking the state of the response, delegate operations or hand over command.

Handoffs and practice

Incidents outlast shifts. The SRE book’s handoff protocol is explicit: the outgoing commander says “you are now the incident commander,” waits for a clear acknowledgment, and tells everyone else on the incident. A handoff with no acknowledgment can leave the incident with two commanders or none. Responders also need a shared working document or channel that captures hypotheses, ruled-out causes, and data collected, because in a fast conversation, findings are lost. Roles are a skill that decays when unused, so exercises should rotate who commands, communicates, and leads operations, as the SRE material recommends.

One more point is an inference from the structure and not a cited claim: the channels used to coordinate the response should not depend on the system that is failing. If the chat system, the ticketing system, or the single sign-on provider runs on the affected platform, the team needs another way to reach each other and a way to log in without it. Decide this and test it before the event.

Runbooks

What a runbook is for

A runbook (the SRE book calls it a playbook) is a written procedure for a known situation. The SRE book’s introduction reports that thinking through and recording best practices ahead of time in a playbook produces roughly a 3 times improvement in mean time to repair compared with improvising, a statement about Google’s experience that is best treated as a reason to try the practice and measure the result yourself. A runbook lowers the cognitive load of an incident, makes the response less dependent on who happens to be on call, and gives a drill something concrete to test.

The SRE monitoring chapter adds the limit. A page should require judgment, and a page whose response is a rote script should be automated. A runbook is for the part that needs a human to decide something. A step that is always the same belongs in code.

What a useful runbook contains

An effective runbook is short enough to follow under stress and specific enough to act on. The structure below is the author’s arrangement of what the sources imply, and not a published template.

1. Trigger. The alert or symptom that leads here, and what it means in user terms.
2. Impact check. How to see who is affected and how badly: which dashboard, which query, which segment.
3. First mitigation options. A short decision list ordered by speed and safety: roll back the last change, disable a feature flag, shift traffic, shed load, scale out, or fail over. Each option states when it applies, the command or procedure, the expected effect, and how long it takes.
4. Verification. How to tell the mitigation worked, using the same SLI that raised the alert, and how long to watch before declaring success.
5. Escalation. Who else to involve and when, including the owner of each dependency.
6. Hazards. What not to do and why: actions that destroy evidence, make data loss worse, or have side effects (sending messages, charging cards).
7. Ownership and review. The owner, the last time someone executed it end to end, and the next scheduled review.

Here is an illustrative skeleton in plain text, for a hypothetical service that exposes a write API in front of a database. It is invented for this article.

Runbook: payments writes producing invariant failures (hypothetical)
Trigger: alert "payments-invariant-check failing" (page) or support report of balance mismatch
Impact check: run the invariant query on the primary (read-only); note first failing minute
Contain first (operations lead only):
  1. Stop new writes: set the write API to read-only mode (flag: payments_writes_enabled=false)
  2. Pause jobs that read the payments table and write elsewhere (settlement export, notifications)
  3. Take a snapshot of the primary and keep it; do not delete or repair anything yet
Decide (incident commander): last-known-good backup vs forward repair (see decision notes)
Verify: invariant query passes; sample accounts reconciled against the independent journal
Escalate: data owner and finance contact if any settled amount is affected
Hazards: do not restore over the primary before the snapshot exists; restoring discards writes after the backup
Owner: platform data team. Last full walkthrough: (date). Next review: (date)

Runbooks decay as systems change. The check is to run them: in an exercise, a new on-call engineer should be able to execute the procedure unaided, and whatever they had to ask about is a defect in the runbook. A runbook whose commands have not been run in a year is a hypothesis about the system as it was.

Containment

Stop the damage first

The SRE incident management chapter’s list of priorities is to stop the damage, restore service, and preserve evidence, and the workbook’s case studies add the working rule: mitigate the impact first and find the root cause afterward. In one case study, generic mitigations such as rolling back or draining traffic, applied early, would have shortened an outage of six and a half hours. The principle is that the cause of an incident is not required before an action that limits it, and waiting for a diagnosis while users are harmed is a choice with a cost.

Containment actions are generic by design: roll back the last release, turn off a feature flag, remove a bad instance from rotation, shift traffic away from a failing zone, shed load, or freeze other changes. They work without understanding the cause, which is why they are fast. Their limits are in what they do not undo. The rollback entry makes the conditions explicit: a rollback helps only if the earlier version still works with current data and interfaces, and it does not unsend a message or reverse an accepted order. Where those effects matter, check before acting.

Data incidents are different

When the incident is damaged data and not unavailable service, the first containment action is to stop the damage from spreading, and that frequently means making the service less available on purpose. Setting writes to read-only, pausing the jobs that propagate the damaged data to other systems, and disabling the release that produced it all reduce harm at the cost of function. This is a trade the commander makes explicitly. A service that keeps accepting writes while corrupting them creates work that has to be repaired later, and any backup taken in that interval contains the corruption, which the demonstration below shows concretely.

Rollback or forward repair

Once the damage is contained, the choice between returning to an earlier state and repairing the current one depends on what has happened since the cutover. Returning is possible only if the restore can carry back the valid work accepted in between and handle external effects; otherwise the choice is forward recovery, correcting the faulty records or behavior with controlled, reviewed changes and verifying the result against requirements. Forward recovery is not uninterrupted: writes may stay paused or limited until the correction is verified. Both paths need an owner, a record of what was changed, and a verification step. The restore demonstration shows both options on the same incident and what each costs.

Preserve evidence while acting

Containment changes the system, and every change risks erasing something the analysis will need. The practical rules are cheap: snapshot or copy affected state before repairing it, keep the logs and metrics for the incident window beyond their normal retention, record each emergency change with its time and author (the planning lead’s job), and route changes through the operations lead so that the record is one list and not several. A containment action that destroys the only evidence of the failure should be the exception that the commander approves knowingly.

Recovery objectives

Definitions

NIST’s contingency planning guide defines the recovery time objective as the overall length of time an information system’s components can be in the recovery phase before negatively impacting the organization’s mission or business processes, and the recovery point objective as the point in time to which data must be recovered after an outage (both via the NIST glossary entries for recovery time objective and recovery point objective). The RTO and RPO glossary entry works through examples. They are separate conditions: meeting one says nothing about the other. The RTO is a limit on how long the service may be unavailable and is met by procedures, people, access, and capacity. The RPO is a limit on how much recent work may be lost and is met by how often and how completely data is captured.

Both are requirements derived from the harm of an outage, not engineering properties to be read off a system. The question to ask the business owner is “what happens if this is down for an hour, a day, a week” and “what happens if the last hour of orders is gone.” The answers differ by service and often by the kind of data within one service, so a platform with one global target is usually either over-spending on some services or under-protecting others.

Achieved versus objective

An objective is a target and a recovery is an outcome, and the two are compared by measurement. The achieved recovery time is the interval from the start of the outage (or from detection, depending on how the objective defines its clock) to the point the service is usable again and has passed validation. It has components that can be measured separately: detection, decision, retrieval or provisioning, restoration or replay, validation, and routing traffic back. A restore of files that finishes in time does not establish that the application can resume the promised work, so the clock stops at a usable service and not at the completion of a copy.

The achieved recovery point is the failure time minus the time of the most recent data actually recovered. A backup’s completion time is not that figure. What matters is the point in time represented by the data in the backup, after checking for gaps and corruption.

A short calculation shows the dependence on schedule. If a consistent copy is taken every 15 minutes and the newest copy cannot be used until it has been verified, which takes 5 minutes, then the data cutoff of the newest usable copy at the moment of a failure is between 5 and 20 minutes old: 5 minutes just after a copy has been verified, and nearly 20 minutes just before the next one is. The worst-case loss is therefore the interval plus the verification delay, 20 minutes, which meets a 30-minute RPO and misses a 15-minute one. Taking copies more often, or adding a log of changes between copies, reduces the exposure. The demonstration below measures this on toy data, including the case in which the newest copy is unusable and the exposure doubles.

Replication is not a backup

Continuous replication copies every change, including the wrong ones. The AWS disaster recovery whitepaper states the consequence: continuous replication has the advantage of being the shortest time to back up data, but it may not protect against data corruption or malicious deletion as well as point-in-time backups. The same paper notes that for a data disaster involving corruption, deletion, or obfuscation, the recovery time will always be greater than zero and the recovery point will always be at some point before the disaster was discovered. A replica gives a low recovery point for loss of a site, and a point-in-time backup is what answers logical damage. The two cover different failure classes, and a design should state which one each protects against.

Strategies and what they buy

The same whitepaper describes four disaster recovery strategies for cloud workloads, in increasing cost and complexity: backup and restore, pilot light, warm standby, and multi-site active/active. The table summarizes the differences it describes, without the specific numbers, which vary by service and implementation.

Strategy	What exists in the recovery location before a disaster	What must happen at recovery	Typical effect on RTO and RPO
Backup and restore	Backups of data, and the means to redeploy infrastructure and code	Redeploy infrastructure, restore data, then route traffic	Longest recovery time; recovery point is the age of the usable backup
Pilot light	Continuously replicated data and a minimal core of infrastructure; application servers switched off	Switch on and scale out application capacity	Shorter recovery time; recovery point depends on replication lag
Warm standby	A scaled-down but fully functional copy that can serve traffic	Scale up	Shorter again; also easier to test continuously
Multi-site active/active	Full capacity serving traffic in several locations	Route around the failed location	Can approach zero recovery time for a site loss; data corruption still needs backups

The decision depends on the RTO and RPO, the failure the strategy must survive (a server, a zone, a region, a data error, an account compromise), and cost. The whitepaper also notes a subtlety relevant to testing: restoring data from backup is a control plane operation, which may not be available during a disaster, so a scheduled periodic restore that keeps a recent operable copy is a mitigation. Failover automation triggered by health checks should be used with caution, because a false alarm incurs the same loss of availability and data as a real event, which is why many designs automate the steps but keep the decision to fail over manual.

Restore tests

A backup is not a recovery

The SRE chapter on data integrity makes the distinction sharply: backups can be loaded back into an application, and archives cannot, and what people want is a restore, not a backup. Its summary of the practice is that you only know you can recover your recent state if you actually do. A successful backup job says that a process ran and exited normally. It does not say that the output is complete, that it is the right data, that the credentials and configuration needed to use it exist, or that the restore takes the time the objective assumes.

The same chapter organizes data loss along three dimensions: root cause (user action, operator error, software bugs, infrastructure defects, hardware, site catastrophes), scope (wide or narrow), and rate (sudden or creeping). The combinations do not share a single defense, which is why it recommends layers: soft deletion so that recoverable windows exist, tiered backups from frequent local snapshots to offsite archives, and out-of-band validation that detects corruption before it propagates. A restore test is how a team finds out which of its layers answers which failure.

A restore demonstration

The program below builds a toy payments table in SQLite and simulates a day of activity in logical minutes. Each minute it books 1,500 transfers, writing a payments row and updating a per-account balance, so that an application-level invariant holds: the balance of each account equals the sum of its payments entries. Every 15 minutes it takes a backup using the SQLite online backup API, which Python exposes as Connection.backup and which works while the database is in use. It also appends every legitimate event to a separate journal file. It then runs three failure scenarios and treats each backup as something to be tested by a restore check: open it, run SQLite’s PRAGMA integrity_check, which verifies the structure of the database and not the application’s meaning, and then run the application invariant.

The program uses Python 3.12 and the standard library only. The data is generated with a fixed seed, so row counts and reported minutes are deterministic, while the timing values depend on the machine and the run. In scenario 2 the newest backup fails and an older one passes. In scenario 3 the invariant fails on the backups taken after minute 100 and passes on the earlier ones.

"""Illustrative restore demonstration (toy data, local SQLite). Logical time is in minutes; each minute the app books EVENTS_PER_MIN transfers."""
import sqlite3, os, random, shutil, time, json, hashlib, sys
random.seed(42)
WORK='demo'; shutil.rmtree(WORK,ignore_errors=True); os.makedirs(WORK)
EVENTS_PER_MIN=1500; BACKUP_EVERY=15; ACCOUNTS=50

def init(db):
    c=sqlite3.connect(db)
    c.executescript("""create table payments(id integer primary key, minute integer not null, account integer not null, amount integer not null);
                       create table balance(account integer primary key, amount integer not null);""")
    c.executemany("insert into balance values(?,0)",[(a,) for a in range(ACCOUNTS)])
    c.commit(); return c

def book(c,journal,minute,n,corrupt=False):
    rows=[]
    for _ in range(n):
        a=random.randrange(ACCOUNTS); amt=random.randint(-50,100)
        rows.append((minute,a,amt))
    if corrupt:   # bad release: writes payments rows with the sign flipped and does not update balance
        c.executemany("insert into payments(minute,account,amount) values(?,?,?)",[(m,a,-x) for m,a,x in rows])
    else:
        c.executemany("insert into payments(minute,account,amount) values(?,?,?)",rows)
        for m,a,x in rows: c.execute("update balance set amount=amount+? where account=?",(x,a))
        journal.write(''.join(json.dumps({'minute':m,'account':a,'amount':x})+'\n' for m,a,x in rows)); journal.flush()
    c.commit()

def check(path):
    """restore test: open, integrity_check, application invariant, report newest minute"""
    t=time.perf_counter()
    try:
        c=sqlite3.connect(f'file:{path}?mode=ro',uri=True)
        ic=c.execute('pragma integrity_check').fetchone()[0]
        if ic!='ok': return dict(ok=False,why='integrity_check: '+ic[:40])
        bad=c.execute("""select count(*) from balance b left join (select account,sum(amount) s from payments group by account) l
                         on l.account=b.account where b.amount<>coalesce(l.s,0)""").fetchone()[0]
        newest=c.execute('select max(minute) from payments').fetchone()[0]
        rows=c.execute('select count(*) from payments').fetchone()[0]
        return dict(ok=(bad==0),why=('invariant violated on %d accounts'%bad if bad else 'ok'),newest=newest,rows=rows,secs=time.perf_counter()-t)
    except Exception as e:
        return dict(ok=False,why=type(e).__name__+': '+str(e)[:50])

def build(fail_minute,bad_from=None):
    db=f'{WORK}/prod.db'
    if os.path.exists(db): os.remove(db)
    jr=f'{WORK}/journal.jsonl'; 
    if os.path.exists(jr): os.remove(jr)
    for f in os.listdir(WORK):
        if f.startswith('bk_'): os.remove(f'{WORK}/{f}')
    c=init(db); j=open(jr,'w'); backups=[]
    for m in range(1,fail_minute+1):
        book(c,j,m,EVENTS_PER_MIN,corrupt=(bad_from is not None and m>=bad_from))
        if m%BACKUP_EVERY==0:
            p=f'{WORK}/bk_{m:04d}.db'; dst=sqlite3.connect(p); c.backup(dst); dst.close(); backups.append((m,p))
    c.close(); j.close(); return db,jr,backups

print('=== Scenario 1: host loss at minute 137, backups every 15 minutes, newest backup healthy')
db,jr,bks=build(137)
os.remove(db)
print('backups on disk:',[m for m,_ in bks])
t=time.perf_counter(); r=check(bks[-1][1]); 
shutil.copy(bks[-1][1],db)
print('restore+checks seconds: %.3f'%(time.perf_counter()-t), r)
print('data lost (minutes of accepted work not in the restored copy): 137 - %d = %d'%(r['newest'],137-r['newest']))
print('journal replay of the minutes after the backup would recover the rest; journal lines: %d'%sum(1 for _ in open(jr)))

print('\n=== Scenario 2: newest backup file is truncated (partial copy), exists with non-zero size')
db,jr,bks=build(137)
newest=bks[-1][1]; size=os.path.getsize(newest)
with open(newest,'r+b') as f: f.truncate(int(size*0.6))
print('newest backup size %d -> %d bytes (file exists, non-zero)'%(size,os.path.getsize(newest)))
for m,p in reversed(bks):
    r=check(p); print(f' backup minute {m:3d}:',{k:v for k,v in r.items() if k!='secs'})
    if r['ok']:
        print(' first verified-good backup: minute',m,'-> achieved recovery point is',137-m,'minutes before the failure'); break

print('\n=== Scenario 3: bad release from minute 101 writes wrong payments rows; detected at minute 137')
db,jr,bks=build(137,bad_from=101)
good=None
for m,p in reversed(bks):
    r=check(p); print(f' backup minute {m:3d}:',{k:v for k,v in r.items() if k!='secs'})
    if r['ok'] and good is None: good=(m,p)
print(' newest backup that passes the invariant check: minute',good[0])
# replay journal (only legitimate events were journaled) on top of the good backup
t=time.perf_counter()
rec=f'{WORK}/recovered.db'; shutil.copy(good[1],rec)
c=sqlite3.connect(rec); n=0
for line in open(jr):
    e=json.loads(line)
    if e['minute']>good[0]:
        c.execute('insert into payments(minute,account,amount) values(?,?,?)',(e['minute'],e['account'],e['amount']))
        c.execute('update balance set amount=amount+? where account=?',(e['amount'],e['account'])); n+=1
c.commit(); c.close()
r=check(rec)
print(' replayed %d journal events after minute %d; check ->'%(n,good[0]),{k:v for k,v in r.items() if k!='secs'}, 'replay+check seconds: %.3f'%(time.perf_counter()-t))
print(' without replay: legitimate work after minute %d is lost = %d minutes'%(good[0],137-good[0]))
print(' note: the defective release also skipped the journal, so the journal ends at minute 100 and cannot supply minutes 101-137.')
print('\n--- Scenario 3b: forward repair of the production copy instead of restoring')
c=sqlite3.connect(db)
before=check(db)
c.execute('update payments set amount=-amount where minute>=101')   # exact inverse of the known defect
c.execute('update balance set amount=amount+coalesce((select sum(amount) from payments l where l.account=balance.account and l.minute>=101),0)')   # the bad release never touched balances
c.commit(); c.close()
r=check(db)
print(' before repair:',{k:v for k,v in before.items() if k!='secs'}); print(' after repair: ',{k:v for k,v in r.items() if k!='secs'})
print(' (valid only because the defect was a known, exactly reversible transformation; the invariant check cannot prove the amounts are right, only that payments and balances agree; the repaired balances were adjusted independently of the payments sums, so agreement is informative)')

=== Scenario 1: host loss at minute 137, backups every 15 minutes, newest backup healthy
backups on disk: [15, 30, 45, 60, 75, 90, 105, 120, 135]
restore+checks seconds: 0.125 {'ok': True, 'why': 'ok', 'newest': 135, 'rows': 202500, 'secs': 0.12383743700047489}
data lost (minutes of accepted work not in the restored copy): 137 - 135 = 2
journal replay of the minutes after the backup would recover the rest; journal lines: 205500

=== Scenario 2: newest backup file is truncated (partial copy), exists with non-zero size
newest backup size 2850816 -> 1710489 bytes (file exists, non-zero)
 backup minute 135: {'ok': False, 'why': 'DatabaseError: database disk image is malformed'}
 backup minute 120: {'ok': True, 'why': 'ok', 'newest': 120, 'rows': 180000}
 first verified-good backup: minute 120 -> achieved recovery point is 17 minutes before the failure

=== Scenario 3: bad release from minute 101 writes wrong payments rows; detected at minute 137
 backup minute 135: {'ok': False, 'why': 'invariant violated on 50 accounts', 'newest': 135, 'rows': 202500}
 backup minute 120: {'ok': False, 'why': 'invariant violated on 50 accounts', 'newest': 120, 'rows': 180000}
 backup minute 105: {'ok': False, 'why': 'invariant violated on 50 accounts', 'newest': 105, 'rows': 157500}
 backup minute  90: {'ok': True, 'why': 'ok', 'newest': 90, 'rows': 135000}
 backup minute  75: {'ok': True, 'why': 'ok', 'newest': 75, 'rows': 112500}
 backup minute  60: {'ok': True, 'why': 'ok', 'newest': 60, 'rows': 90000}
 backup minute  45: {'ok': True, 'why': 'ok', 'newest': 45, 'rows': 67500}
 backup minute  30: {'ok': True, 'why': 'ok', 'newest': 30, 'rows': 45000}
 backup minute  15: {'ok': True, 'why': 'ok', 'newest': 15, 'rows': 22500}
 newest backup that passes the invariant check: minute 90
 replayed 15000 journal events after minute 90; check -> {'ok': True, 'why': 'ok', 'newest': 100, 'rows': 150000} replay+check seconds: 0.485
 without replay: legitimate work after minute 90 is lost = 47 minutes
 note: the defective release also skipped the journal, so the journal ends at minute 100 and cannot supply minutes 101-137.

--- Scenario 3b: forward repair of the production copy instead of restoring
 before repair: {'ok': False, 'why': 'invariant violated on 50 accounts', 'newest': 137, 'rows': 205500}
 after repair:  {'ok': True, 'why': 'ok', 'newest': 137, 'rows': 205500}
 (valid only because the defect was a known, exactly reversible transformation; the invariant check cannot prove the amounts are right, only that payments and balances agree; the repaired balances were adjusted independently of the payments sums, so agreement is informative)

Reading the results

Scenario 1: the healthy case produces measurements. With the database lost at minute 137 and backups every 15 minutes, the newest backup is from minute 135. The restore plus checks took about 0.12 seconds in the run shown (a second run took 0.116) on this toy database of 202,500 rows. The recovered position is minute 135 and the achieved recovery point is 2 minutes. That number is lucky: failures at minute 134 would have lost 14 minutes, and the objective has to be judged against the worst case, 15 minutes plus verification, and not against one outcome. The restore time is a real measurement but of a database that fits in memory. Real restore time scales with data volume, network and storage throughput, and the work after the copy (replay, index validation, routing), so the figure has to be measured at production scale in a representative environment before it informs an RTO.

Scenario 2: the file exists and is useless. The newest backup file was truncated to 60 percent of its size, from 2,850,816 to 1,710,489 bytes. A check for “file exists and is not empty” passes. The restore check fails with “database disk image is malformed”. The next older backup, minute 120, passes, so the achieved recovery point is 17 minutes instead of 2. The objective was met or missed depending on whether anyone had tested the newest copy, and a team that tests only the backup job’s exit status would have learned this during the incident. The practical conclusions are to verify every backup by restoring it (or a sample, with a stated coverage), to alert when the newest verified backup is older than the RPO allows, and to measure the age of the newest verified backup, which is the figure that actually bounds the recovery point.

Scenario 3: the newest backups are the wrong ones. A defective release begins at minute 101 and writes payments rows with the sign flipped, without updating balances. The service keeps running, and each backup after that point copies the damage. Detection happens at minute 137. The invariant check fails on the backups from minutes 105, 120, and 135, and passes on those from minute 90 and earlier. The newest usable backup is therefore minute 90, which is 47 minutes before detection. Without more information, 47 minutes of legitimate work would be lost.

The pattern also yields evidence for the analysis, which the last section returns to: the last passing backup is minute 90 and the first failing one is minute 105, so the corruption began between minute 91 and minute 105, consistent with the release at minute 101. A set of verified backups acts as a bisection over time.

The journal was meant to close the gap. Replaying its events on top of the minute-90 backup recovered 15,000 events and reached minute 100, where the journal ends, and passed the check. It could not supply minutes 101 to 137, because the defective release also skipped the journal. The independent record was not independent of the defect, since it was written by the same code. A log that is meant to survive a bug in the writer has to be produced by something the bug cannot change, such as a separate capture of the incoming events or a write-ahead log maintained by the database engine and not by the application.

Scenario 3b: forward repair. Instead of restoring, the demonstration repairs the production copy by reversing the known defect: it negates the payments amounts from minute 101 onward and adds the corrected amounts to the balances, which the defective release had never touched. After the repair the invariant passes and no minutes are lost. The repair was possible only because the defect was a known, exactly reversible transformation. The invariant check shows that payments and balances agree and cannot prove the amounts are right; here the agreement is informative because the balances were adjusted from the corrected entries independently of any recomputation from the payments table. In an incident the safer sequence is usually to verify the repair on a copy first, keep the pre-repair snapshot, and have a second person review the repair, which is why the runbook skeleton above takes a snapshot before changing anything.

The limits of the demonstration: toy data, a single database file, a defect that is simple and total, and an invariant that happens to exist. Real systems rarely have a check this clean, and building a meaningful validation query for each important dataset is much of the work. The demonstration does show the shape of a restore test: restore into a scratch location, check structure, check meaning, record the recovered position, and record the time.

Point-in-time recovery in a real database

For a production relational database, point-in-time recovery combines a base backup with a continuous archive of the write-ahead log. The PostgreSQL documentation states the central requirement: you need a continuous sequence of archived WAL files that extends back at least as far as the start time of the base backup, and a gap breaks recovery past it. A sketch of the configuration from the documentation, shown here for orientation and not executed for this article, looks like this:

# postgresql.conf, archiving (sketch from the PostgreSQL documentation; adapt paths and commands)
wal_level = replica
archive_mode = on
archive_command = 'test ! -f /mnt/server/archivedir/%f && cp %p /mnt/server/archivedir/%f'

# at restore time: create recovery.signal in the data directory and set
restore_command = 'cp /mnt/server/archivedir/%f %p'
recovery_target_time = '2024-01-15 14:30:00'

Three properties from the documentation matter for planning. Each completed archive recovery creates a new timeline, so the history after a recovery is distinguishable from the history before it. Configuration files such as postgresql.conf and pg_hba.conf are not part of the archive and have to be backed up separately, which is a typical hidden dependency of a restore. And the documentation stresses testing: verifying that archiving succeeds is not the same as verifying that recovery works, so recovery itself has to be exercised. Point-in-time recovery shortens the recovery point to seconds. It does not shorten the recovery time, since replaying a long log takes time, and the point-in-time recovery entry covers that distinction.

A schedule for restore tests

A useful program has a few elements, which are the author’s recommendations from the sources above and not a standard.

• Automated, regular restores into a scratch environment, each followed by structural checks, application-level checks, and a record of how long it took and which recovery point it reached.
• An alert on the age of the newest verified backup, tied to the RPO.
• Scenario coverage across failure classes: loss of the primary, a truncated or corrupted backup, logical corruption found late, an accidental deletion, and loss of a whole failure domain. Each of these is answered by a different layer.
• Representative scale, at least periodically, because time and failure modes change with volume.
• A restore path that does not depend on the failed system: credentials, keys, configuration, tooling, and runbooks should be reachable if the primary environment and its identity provider are gone.
• A record of results that feeds the RTO and RPO claims, so that objectives are backed by measurements.

Disaster exercises

Kinds of exercise

An exercise tests the response and not only the technology. The author’s arrangement of the practical range, from least to most costly, follows, and the choice depends on what you need to learn.

Kind	What it tests	Cost and risk
Walkthrough or tabletop	Roles, decisions, communication, and whether the runbook makes sense, discussed against a scenario	Low; discovers gaps in plans and knowledge but proves nothing about the technology
Restore drill	A restore or failover of a component into a scratch environment, timed and validated	Moderate; produces measured recovery times and points
Fault-injection game day	Detection, response, and recovery with a real fault in a controlled scope (see the previous article)	Higher; needs a limited blast radius and abort conditions
Full failover exercise	The complete move of production work to the recovery location	Highest; may itself cause an outage if the plan is wrong

The recovery drill entry quotes the SRE book’s position that components that are not continually exercised fail when they are needed most, and that recovery that has not been tested regularly will not work. The practical reading is that an untested procedure is a hypothesis about a system that has since changed, and an exercise converts it into a fact in whichever direction the fact goes.

Designing an exercise

Define the scenario and what is in scope. State the success criteria in advance, in terms that can be measured: the recovery time and point against the objectives, whether the runbook steps worked as written, whether communication reached the right people, whether the people who needed access had it. Rotate the roles so that more than one person can command, communicate, and lead operations. Assign observers who record what happened, with times, and do not take part. Set abort conditions, so that an exercise that is harming production is stopped. And decide before the exercise who tracks the action items afterward.

The SRE book describes Google’s February 2011 Gmail incident, the first large-scale use of its global backup system to restore live customer data, and notes that similar situations had been simulated many times before. As the book reports it, those rehearsals allowed the team to estimate how long the restore would take and to complete it within several hours of that estimate. The transferable claim is narrow: measured rehearsals give a basis for an estimate and for communication with stakeholders, which untested plans do not. It does not mean that rehearsal guarantees an outcome.

An exercise scorecard, with invented numbers, shows what to record:

Item	Objective	Measured in exercise	Result
Time from failure injection to detection	5 minutes	4 minutes	Met
Time from detection to incident declared	10 minutes	14 minutes	Missed; criteria were unclear
Restore of the payments database to a usable service	60 minutes	83 minutes	Missed; replay and validation were not in the estimate
Recovery point	15 minutes	17 minutes	Missed; newest backup failed verification
Runbook steps executed as written	all	6 of 9	Three steps referred to retired tooling

Each miss turns into an action item with an owner and a date. The exercise has succeeded when it finds these before a real incident does.

Evidence-based incident analysis

Blameless, and specific

A postmortem records an incident’s impact, response, contributing causes, and follow-up work. In Google’s blameless approach, as the SRE postmortem chapter describes it, the analysis assumes people acted in good faith with the information they had, and focuses on how the system and process allowed the failure, on the reasoning that you cannot fix people but you can fix systems and processes. The blameless postmortem entry covers why this is a mechanism for getting accurate disclosure and not a courtesy. Blameless does not mean vague: the document still names the actions, the evidence, and the decisions, and it examines the conditions in which they were taken.

The chapter lists example triggers for writing one: user-visible downtime or degradation beyond a threshold, any data loss, an on-call intervention such as a rollback or a traffic reroute, a resolution time above a limit, and a monitoring failure that required manual discovery. The thresholds are for each organization to set. It also says that an unreviewed postmortem might as well never have existed, which is a statement about follow-through: the document must be reviewed, and its action items must be tracked to completion.

Build the timeline from sources

Evidence-based analysis starts with a timeline in which every entry has a timestamp, a source, and a label for whether it is an observation or an inference. Sources include deployment records, alert history, metrics, logs, chat transcripts, the incident’s working document, and the records of emergency changes. Disagreements between sources (clock skew, time zones, logs with delayed ingestion) are noted and not smoothed over. People’s recollections are useful for decisions and reasoning but are weak evidence for times.

The table below is a hypothetical timeline for the demonstration’s scenario, in the demonstration’s logical minutes. The demonstration produced only the facts in the first and second rows (release at 101, failure checks later); the detection, declaration, containment, and repair entries are invented to show the form.

Minute	Event	Source	Type
101	Release begins serving traffic; payments rows written with flipped sign	Deploy record; payments rows	Observation
90 and 105	Last passing and first failing backup	Restore checks	Observation
91 to 105	Window in which the corruption began	Derived from the two checks above	Inference
137	Customer reports a balance mismatch	Support ticket (hypothetical)	Observation
139	Incident declared; commander named	Incident channel (hypothetical)	Observation
142	Writes set to read-only	Operations log (hypothetical)	Observation
162	Repair applied to a copy and verified	Operations log (hypothetical)	Observation
170	Writes re-enabled after validation	Operations log (hypothetical)	Observation

Measure the intervals

From a timeline like this, define each interval with its start and end events, because the same word covers several measurements. In the hypothetical, the time to detect, from the start of the defect to the first detection, is 137 − 101 = 36 minutes. The time from detection to declaring the incident is 2 minutes. The time from detection to containment is 5 minutes. The time from detection to a validated recovery is 33 minutes. The interval during which the data was being damaged is 101 to 142, which is 41 minutes. These figures describe one incident under one set of definitions and are not benchmarks. The useful reading is that the largest component, 36 minutes, is detection, which a faster response or a better runbook would not have shortened. The finding that the invariant check ran only at restore time and not continuously leads to a different action item (run it as a monitor) than a finding that the response was slow would.

Contributing factors, not a single root cause

The analysis asks why each condition allowed the failure and which conditions together were sufficient. In the hypothetical: the release changed the write path without a test for the payments invariant; the journal was written by the same code as the payments table and so was not independent; no monitor checked the invariant continuously; the backup job’s success was judged by its exit status. None of those is a person, and each leads to a checkable change. Resist stopping at the last human action in the chain, because that is where the analysis ends and learning stops.

Separate what is known from what is inferred, and say what could not be determined. A statement such as “the corruption began between minute 91 and minute 105” is an inference from two observations and it is stated with its support. A statement such as “the release caused the corruption” requires evidence beyond timing, for instance reproducing the defect from the release’s code path, and the analysis should say which kind of support it has. Be careful of hindsight: the question is what was knowable at each decision point with the information then available.

Actions that close the loop

Each action item is specific, has an owner and a due date, has a priority, and is tracked. Prefer items that change the system (add a continuous invariant monitor, put the journal outside the application’s write path, alert on the age of the newest verified backup) over items that ask people to be more careful. Tie the incident to the objective: how much of the error budget from the earlier article it consumed, and whether it triggers a rule such as an automatic postmortem for incidents above a given share. Review the document with someone who was not involved, and check later whether the actions were completed and whether they worked, for example by repeating the failure in an exercise.

Failure modes of incident programs

• No one is in charge, or two people are. The remedy is explicit assignment and acknowledged handoffs.
• Responders change production in parallel. Funnel changes through one role and keep a single list.
• Runbooks that have not been executed. Run them in exercises and track the date of the last end-to-end execution.
• Backups that were never restored. A green backup job is evidence that a job ran. Restore and validate, and monitor the age of the newest verified backup.
• Backups in the same failure domain as the data. If the same credentials, account, or storage system can lose both, the backup protects against fewer failures than it appears to. Review which failure classes each copy survives.
• Replication treated as backup. It propagates corruption and deletion.
• A recovery time that was never measured. Estimates built from component timings leave out detection, decision, validation, and routing, which exercises routinely show to be significant.
• Containment that destroys evidence or widens the damage. Snapshot first, and check the side effects of every emergency action.
• Communication neglected. A response with no communications role has engineers writing updates, or no updates.
• Postmortems with no follow-through. The document is not the outcome. The completed, verified actions are.
• Exercises that are scripted to pass. An exercise whose outcome is known in advance teaches little. Vary the scenario, include surprises within safe limits, and rotate participants.

A checklist

1. Write down the criteria for declaring an incident, and the four roles with their alternates, and rehearse the handoff phrase.
2. Keep runbooks short, with a trigger, an impact check, ordered mitigations, verification, escalation, and hazards, and record when each was last executed.
3. Automate steps that never need judgment.
4. For data incidents, define the first containment steps (stop writes, pause propagation, snapshot) before they are needed.
5. Set an RTO and an RPO per service from business impact, define when each clock starts and what counts as usable, and keep the objectives separate.
6. Back up against logical damage as well as site loss, and state which failure classes each copy covers.
7. Restore regularly into a scratch environment, validate structure and meaning, and record the time and the recovery point reached.
8. Alert on the age of the newest verified backup.
9. Run exercises of increasing cost, with measurable success criteria, rotating roles, and tracked action items.
10. Analyze incidents from a sourced timeline, separate observation from inference, measure intervals with stated definitions, and track actions to completion.

Limits and next steps

What this article establishes: the structure of an incident response (declaration, roles, handoffs, runbooks, containment), the definitions and arithmetic of recovery objectives, the difference between a backup and a verified recovery as demonstrated on toy data with a stated invariant, and the structure of an evidence-based analysis. What it does not establish: the right RTO, RPO, backup interval, or exercise frequency for your service, which depend on the harm an outage causes and the cost you can bear, or that any of these practices improves outcomes in a given organization. The demonstration is a model with a single database, a simple defect, and a clean invariant, and the incident timeline is hypothetical.

Reliability has a price, in capacity held in reserve, copies kept, exercises run, and engineer time. The next article, Platform Unit Economics and Cost Controls, takes up how to attribute those costs and decide how much reliability to buy.

References

• Google, Site Reliability Engineering, Managing Incidents
• Google, Site Reliability Engineering, Postmortem Culture: Learning from Failure
• Google, Site Reliability Engineering, Data Integrity: What You Read Is What You Wrote
• Google, Site Reliability Engineering, Introduction
• Google, Site Reliability Engineering, Monitoring Distributed Systems
• Google, The Site Reliability Workbook, Incident Response
• NIST CSRC Glossary, recovery time objective
• NIST CSRC Glossary, recovery point objective
• AWS Whitepaper, Disaster recovery options in the cloud
• PostgreSQL documentation, Continuous Archiving and Point-in-Time Recovery
• Python documentation, sqlite3 (Connection.backup)
• SQLite documentation, PRAGMA integrity_check