Recovery Drill

A recovery drill is a deliberate rehearsal of a restore or a failover, run when nothing is wrong, to establish that it works and how long it takes. Organizations call it a disaster recovery test, a DR drill, a game day, or — at Google — a DiRT exercise. The name matters less than the principle behind it, which the SRE book states as a heading: “Hope Is Not a Strategy.”

System components that aren’t continually exercised fail when you need them most. Prove that data recovery works with regular exercise, or data recovery won’t work.

Read the second sentence as a claim about knowledge rather than about diligence. An unexercised recovery procedure is a hypothesis: it describes what someone believed would happen when they wrote it, against a system that has since changed. A drill tests that hypothesis for the scenario, environment, and data it actually exercised, and the result can go in either direction. It is evidence about that scope, not proof that the whole procedure works under every failure.

What rehearsal actually buys

The documented case is Google’s February 2011 Gmail restore, “the largest use of our last line of defense, the GTape offline backup system.” The key clause is about what had happened before the incident: “Fortunately, it was not the first such restore, as similar situations had been previously simulated many times.” Three results followed.

1. “Deliver an estimate of how long it would take to restore the majority of the affected user accounts“
2. “Restore all of the accounts within several hours of our initial estimate“
3. “Recover 99%+ of the data before the estimated completion time“

Rehearsal provides measured inputs for a recovery estimate, including throughput, dependency order, and handoff time. Communications can use an estimate with an explicit range and assumptions; support and business owners can weigh temporary alternatives. Teams without a representative restore history may still estimate from component tests or provider evidence, but should disclose what remains untested rather than present that estimate as a demonstrated recovery time.

Google attributes the Gmail outcome to planning, engineering work, and coordination. The SRE incident-management chapter also recommends practicing response roles as part of disaster-recovery testing because rarely used skills decay. Include the people responsible for decisions and communication as well as those running the restore commands.

What counts as passing

We restored something is not a result. Six criteria make a drill decidable.

• Inside the time objective. Define the clock’s start and the usable-service condition before the exercise. Include the required retrieval, provisioning, replay, validation, routing, and access checks under representative data volume and resources. If the target is detection-based, record prior detection delay separately. A file restore that finishes in time does not prove the application can resume the promised work. A scenario exceeding its time target is a miss even if the exercise produces valuable learning.
• The data is right. Restored is not the same as correct, so check it independently — “check the correctness of the most critical elements of your data using out-of-band data validators, even if API semantics suggest that you need not do so,” because “perfect algorithms may not have perfect implementations.”
• Inside the recovery-point objective. Record the recovered data position and missing accepted work, then compare them with the RPO. Yesterday’s backup might open quickly and pass structural checks while missing a day of orders. A completeness or freshness validator with an appropriate reference can detect that gap; a schema or internal-consistency check alone cannot. Specify what each check establishes rather than treating all validation as interchangeable.
• The procedure carried it, not a person. If the drill succeeded because the one engineer who remembers the system was in the room, you have tested that engineer’s availability.
• Everything needed was reachable. Credentials, keys, the runbook itself, the account holding the backups — under the conditions of the scenario, not under normal conditions.
• A controlled return to service. Name who can stop the exercise, the impact limits that trigger a stop, and the recovery path if a step fails. For a live failover, account for writes accepted at the new site before failing back: reconcile or replicate them and prevent conflicting writers. Returning traffic to an old endpoint is not enough. An isolated restore also needs cleanup and controls against sending real notifications or other unintended external effects.

The gaps a first drill usually finds are mundane and specific: a credential that expired, a decryption key held by someone who left, a backup whose schema the current code cannot read, a restore that takes as long as the backup did because nobody had measured it, and a dependency order nobody wrote down — the service that cannot start until another service is already up. All cheap discoveries on a Tuesday.

Scaling up without breaking production

Build confidence with progressively broader exercises. The sequence below is a practical starting approach, not a universal maturity ladder; select a scope that the team can observe, stop, and recover from.

1. Read the procedure aloud. Catches steps that reference systems that no longer exist, and people who no longer work here. Nearly free.
2. Tabletop the scenario. Walk a specific failure through the plan with the people who would be involved, deciding rather than reciting. Catches missing decisions and unclear authority — and this is where role-playing helps: teams “often role-play the response to an on-call issue that has already been solved, perhaps by colleagues in another location.”
3. Restore for real, to somewhere else. Load an actual backup into an isolated environment, validate it, and record elapsed time and resources. Apply the six criteria above to the declared scenario and mark any untested conditions. A small sample proves a narrower capability than a full-volume restore and does not establish production RTO or failover capacity.
4. Exercise the failover itself. For active/passive designs, invoke it. For designs with no failover to invoke, the target changes: with multi-site active/active “there is no such thing as failover in this scenario,” so testing “would focus on how the workload reacts to loss of a Region: Is traffic routed away from the failed Region? Can the other Region(s) handle all the traffic?“

One warning about the last level. Running in several places is not a substitute for drilling restores, because a topology test and a data test check different failures: “Testing for a data disaster is also required.” A corrupted row is present in every region by design.

Worth noting that the architecture affects how easy this is. A warm standby arrangement “allows you to more easily perform testing or implement continuous testing to increase confidence in your ability to recover from a disaster” — so the money spent on a standing environment buys rehearsal as well as recovery time, which is an argument that rarely makes it into the cost comparison.

Frequency, automation, and the thing that decays

The honest answer to “how often” is: often enough that nothing important has changed since the last one. The principle behind it is stated as a warning about time rather than about discipline — “The fact that your data ‘was safe yesterday’ isn’t going to help you tomorrow, or even today. Systems and infrastructure change, and you’ve got to prove that your assumptions and processes remain relevant in the face of progress.” A continuity plan on the same footing: “business continuity planning is a process, not a one-time event,” to be “reviewed and updated regularly.”

Which argues for automating the frequent part: “humans lack discipline to continually exercise system components, so automation is your friend.” With a caveat the source adds and most teams live through — “when you staff such automation efforts with engineers who have competing priorities, you may end up with temporary stopgaps.” An automated nightly restore of a sample, plus a human-run full drill on a slower cadence, is the arrangement that survives contact with a roadmap.

Prefer well-understood mechanisms with a substantial operational history, as the SRE book recommends. That evidence can reduce uncertainty about the implementation, but it does not test your backup contents, keys, permissions, workload scale, or restoration procedure. Established tooling and local rehearsals address different sources of uncertainty.

Report whether each recovery objective passed, failed, or remained untested, separately from what the exercise taught the organization. Discovering a broken step is useful learning but does not turn a missed RTO into a pass. Start with an announced, bounded exercise involving the relevant stakeholders and a control team with authority to stop it. If a later blind exercise is justified, obtain approval and prepare observers, impact limits, stop conditions, and return procedures; specify which details are withheld from participants while the control team remains informed. AWS’s game-day guidance emphasizes preparation and stakeholder notification, especially for early exercises. Record corrective actions and retest them. Replication Is Not Recovery explains the surrounding recovery plan.

The question to hold your own program against: when did you last restore from your oldest and least-used layer, how long did it take, and who could do it if the person who did it last time were unavailable? None of those can be answered from a document.

References

• “Data Integrity: What You Read Is What You Wrote,” Google SRE Book
• “Managing Incidents”
• AWS, Disaster recovery options in the cloud
• Microsoft Learn, Business continuity, high availability, and disaster recovery
• AWS Well-Architected — Conduct game days regularly