Disaster Recovery and Recovery Objectives

Google’s SRE book describes a flawed design-review response: a team asked about backups points to replication as its supposedly better protection.

Automatically synchronized replicas can propagate a corrupted row or mistaken deletion to every copy before the team isolates the problem. Replication can preserve availability while reproducing the data loss.

Replication keeps copies aligned, including when an application issues a DELETE with a missing WHERE clause. Google’s SRE book illustrates the consequences with an email provider that suffered a ten-day outage, initially announced data loss, and later reported that the information could be recovered. Preserved data was unavailable to the people who needed it for too long. From their perspective, that interruption could be as damaging as loss; the case does not establish which failure type is most common across companies.

Recovery depends on more than aligned replicas. Copies need protection from logical corruption, a purpose and retrieval process that fit the recovery requirement, and procedures whose performance has been demonstrated. The following cases show where replication, long-term retention, and documentation alone can fall short.

Two numbers, and where they come from

Recovery planning rests on two targets — how long a service may be down, and how much recent data may be lost. Their definitions and arithmetic are in recovery time objective and recovery point objective; what matters here is where the numbers come from, because most organizations pick them by intuition and then discover what they implied.

Microsoft’s current guidance begins by classifying workloads by business criticality, then identifies risks, classifies them, and chooses mitigations. Start with the processes that must continue, the harm caused by interruption or lost accepted work, and the dependencies needed to resume them. Business owners and engineers can then agree recovery objectives and assess whether a proposed design can meet them. Risk controls include technical mechanisms and changes to people or procedures.

The HA/DR distinction depends on the workload as well as the event’s likelihood and impact. A failed machine may be handled by normal availability mechanisms. A regional outage may require disaster recovery for a single-region workload, while a suitably provisioned multi-region active/active design can treat it as an HA event. Identify what the current architecture survives automatically and what requires recovery; neither frequency nor the name of the failed component settles the classification alone.

After a 2011 incident, the SRE authors judged four days of recovery potentially too long and proposed 24 hours as an initial threshold for Google Apps. That was a starting point for a consumer application at that scale, to debate against its needs rather than adopt universally.

Availability and data integrity use different denominators. The SRE chapter compares 99.99% annual uptime, allowing roughly an hour of downtime, with 99.99% correct bytes in a 2 GB artifact, allowing about 200 KB of damage. The latter can destroy an executable through changed opcodes or make a database unloadable; a high percentage alone does not establish usable data.

Set uptime and data integrity requirements separately, even where either was previously implicit. Agree them with users before data is lost. Setting reliability objectives develops how reliability targets are negotiated; recovery planning also needs explicit requirements for the data itself.

Three scopes that get used interchangeably

Business continuity is the wider concern. High availability and disaster recovery contribute to it through related strategies; they are not a single chain in which one contains the other. Microsoft’s definitions distinguish their roles:

• Business continuity maintains business operations during failures, outages, and disasters through advance preparation and resilient systems and processes.
• High availability designs for routine operational faults while meeting the business’s availability needs.
• Disaster recovery plans responses to unusual risks and the major outages they can cause.

Continuity covers the cloud platform and application as well as people and manual or automated business processes. A runbook that depends on an unreachable person can fail even when the technical recovery path remains available.

Continuity planning requires stakeholders to agree expectations and consequences, assess the costs, and revise the proposal around budget and other constraints. Review and update that agreement regularly as those conditions change. Business continuity covers the wider frame.

Why replication does not recover you

Replication places data where it is needed and can protect against a site-specific disaster. It cannot handle every kind of data loss: the same propagation that maintains redundant copies can spread a mistaken update or deletion. Recovery needs protection against those scenarios too.

Frequent exports in a native file format can preserve a separate recovery copy against user mistakes and application bugs. They remain vulnerable to failures below that boundary. In the source’s zero-day example, a compromised filesystem or device driver threatens every copy that depends on it, including exports stored on the database’s own filesystem.

Choose diverse components at the layer whose failure you need to survive. Media isolation can protect against media-specific faults: a disk-driver defect or attack is less likely to affect tape drives. The authors’ clay-tablet joke pushes the same idea to an extreme—preserve valuable data using a medium that does not share the compromised dependency.

That framing is what makes the logical-corruption and ransomware cases tractable. An attacker or a bug with access to your primary storage has access to everything reachable from it through the same credentials, filesystem and API. A copy is only a defense if the thing that could corrupt the original cannot reach it — which is a statement about isolation, immutability and separate authority, not about copy count. Three replicas in three regions under one control plane are one failure domain for this purpose.

The SRE book illustrates recovery tradeoffs with transaction replication taking seconds, a database export taking 40 minutes, and a filesystem backup taking hours. These are examples of particular mechanisms, not a conversion from copy duration to RPO. A snapshot representing 10:00 that finishes copying at 10:40 is already 40 minutes old when it becomes available; if it is the newest usable copy at an 11:00 failure, an hour of subsequent changes is outside it. This timeline is illustrative. Record the copy’s consistent data position, capture interval, transfer or archive lag, and the failure time. Measure retrieval and replay separately when estimating recovery duration.

Isolation determines which failures a copy can survive; it does not impose a fixed freshness penalty. Frequent log shipping to a separately controlled repository can combine isolation with recent recovery points, while offline media may trade freshness and retrieval speed for a different failure boundary. Compare the actual protection, recoverable position, recovery duration, and cost of each layer.

A backup is not an archive

Distinguish the purpose a copy was built for from whether it is a recovery mechanism this service can rely on. These are two independent axes. On the purpose axis, an archive is kept to answer an auditor or a retention rule, and its retrieval time was never meant to fit inside an outage. On the reliance axis, what matters is measured: how long retrieval plus restore actually takes against the objective, whether the software you run today can read that format, and whether anyone has done it recently. The two axes are independent. Perfectly loadable backups are routinely parked in cold, cheap storage — cloud tape-gateway services exist precisely to retrieve an archived virtual tape and restore from it — so a cold tier does not make a copy an archive.

The SRE chapter distinguishes backups intended for loading back into an application from archives intended for long-term retention. Assess that intended role alongside measured recovery performance: it does not mean every archival format is technically unloadable.

Archives preserve records for long-term audit, discovery, and compliance, with retrieval deadlines suited to those purposes. The source’s example retains seven years of financial transactions; taking a week to recover logs during a month-long audit may be acceptable. An operational backup instead needs to restore service data quickly enough to fit that service’s availability needs.

And the converse matters more, because it is where the comfortable answer hides. A backup that nobody has restored, or whose restore takes three days against a four-hour objective, does not become an archive; calling it one would be a promotion it has not earned. It is a backup that is unverified, or one that does not meet the requirement, and those are the words to write in the record — because each points at different work. An unverified backup needs a drill. One that is too slow needs a different recovery strategy, or a renegotiated objective. Reclassifying it fixes neither.

Plan backward from the restore users need. Creating backup copies is the means; successful recovery is the outcome. Recovery requirements should determine the backup design, rather than treating the existence of a copy as completion.

Choose the recovery scenarios before choosing the backup tool or schedule. The scenarios determine the methods, full or incremental capture frequency, storage locations, and retention period. Recent data-loss allowance, retrieval speed, and the oldest damage you may need to undo create different requirements:

• How much recent data can you lose? Tighter loss limits increase the need for incremental backups. An older Gmail version used near-real-time streaming backups. Frequent full backups also consume live datastore compute while users are being served, bringing the service closer to its capacity and performance limits.
• How quickly must you recover? Shorter rescue times favor nearby recovery copies. The documented tiers keep costly, fast snapshots briefly inside the storage instance, older copies on nearby distributed storage, and longer-retained copies at nearline or offline locations. Local tiers alone do not cover a site-level failure.
• How far back must you reach? Google found gradual application mutation or deletion bugs that were noticed months after loss began, requiring distant recovery points. Rapid code and schema changes pull the other way: old copies can become costly or impossible to use.

Slow, quiet mutation can require old recovery points just as rapid schema changes make them harder to use. Detection speed therefore belongs in the recovery strategy. Point-in-time recovery can help where the datastore supports it. The chapter recommends balancing point-in-time recovery with tiered backups when needed, rather than omitting both.

Layers, because no single mechanism covers the cases

No single mechanism covers every combination of data-loss failures. Use multiple layers so that a later defense can handle less common scenarios beyond the earlier layer’s protection.

Layer	Catches	Documented detail
Soft deletion	Accidental deletion by users, by add-ons, and by hijackers	Protects against inadvertent deletion by delaying removal, commonly for 15, 30, 45, or 60 days
Backups and recovery	Loss beyond what the application itself can undo	Provides the next recovery layer after soft deletion
Regular data validation	Low-grade corruption nobody has noticed yet	Independent checks for defects in implementations, even where the underlying algorithm is correct
Replication, across all layers	Site loss, latency, bandwidth	Can assist selected recovery scenarios, but does not replace a data recovery plan

Google describes application deletion bugs as a major source of loss in the environment discussed, making a bounded undelete capability an important early defense. Its observation that most account hijacking and data-integrity issues were detected within 60 days informed its retention discussion; it is not a universal maximum or a legal deletion rule. Choose a window using local detection delays, late-case losses, storage cost, and applicable obligations. See soft deletion for the marking, recovery, and eventual destruction responsibilities.

Backups need redundancy too: the needed data may itself be missing, or the site holding the best copy may be unavailable for maintenance. Choose a mature redundancy scheme exercised continuously by many users, rather than one whose only evidence comes from the team’s rare restore attempts.

What the strategies cost

AWS groups infrastructure recovery into four approaches, progressing from lower-cost, simpler backups to more complex deployments across active Regions. In active/passive designs, the passive site begins serving traffic only after failover.

Strategy	What is standing	Cost and recovery
Backup and restore	Data copies only; infrastructure must be rebuilt	Lowest-cost approach. Recreate infrastructure, configuration, and application code in the recovery Region as well as restoring data
Pilot light	Data replicated and core infrastructure provisioned; application servers not running	Limits recurring recovery cost by keeping fewer resources active
Warm standby	A fully working production copy running at smaller capacity	Reduces recovery delay because the workload is already running in another Region
Multi-site active/active	The workload runs in several regions, all serving traffic; surviving capacity must be sized and tested	Highest complexity and cost; can approach zero recovery time for many disasters

Pilot light and warm standby both retain primary assets, but they differ at the first request after failure. Pilot light needs servers started, possibly additional non-core infrastructure deployed, and capacity increased before it can serve. Warm standby can already handle traffic at reduced capacity and needs scaling. Choose between them using the workload’s RTO and RPO.

Code, configuration, infrastructure definitions, and the credentials needed to restore them belong in the recovery plan alongside user data. AWS recommends infrastructure as code to make redeployment repeatable. It also recommends minimizing control-plane dependencies for maximum resilience, but this is a design tradeoff: backup/restore and pilot-light strategies need provisioning, and a scaled-down warm standby may need Auto Scaling. Preprovisioned capacity can serve an agreed load without that expansion; cheaper standby capacity accepts the dependency. Record the initial load the recovery site can handle, quotas and resources needed to grow, and what happens if scaling is unavailable. Active/active likewise requires evidence that the remaining regions can absorb the failed region’s work.

Multi-site active/active can approach zero recovery time for many disasters, but logical corruption can still require restoration from backup with a nonzero recovery point. Backup and recovery remain necessary, and tests must include data corruption as well as regional loss. A topology that propagates updates across every site can also propagate wrong data across every site.

Prove the recovery capability

Turn the plan into a demonstrated capability by exercising recovery regularly. Unused components may fail when finally needed. Automation helps sustain those exercises when people cannot reliably repeat them by hand.

In February 2011, Gmail used its GTape offline system for its largest last-line recovery at that time. Repeated earlier simulations helped the team estimate when most affected accounts would be restored. It restored all accounts within several hours of that initial estimate and recovered more than 99% of the data before the estimated completion time.

Rehearsal gives an incident team measured evidence for its estimate, including restore throughput and coordination delays. That helps communications, support, and business owners decide how to respond. Without a representative restore history, an estimate can still use component measurements or provider evidence, but its uncertainty and untested assumptions should be explicit. Google’s case attributes its forecasting and recovery performance to preparation and cooperation.

Public surprise that Google had recovered from tape highlighted the value of diverse defenses. The incident exceeded what Gmail’s internal redundancy and backup mechanisms could recover. Tape also supplied protection against wider faults or zero-day vulnerabilities in disk drivers and filesystems. Recovery drill explains which scenarios a rehearsal should cover and what establishes a pass.

Yesterday’s answer expires

Recheck recovery assumptions as infrastructure and systems change. Yesterday’s successful protection does not establish that today’s dependencies, data, and procedures remain recoverable.

The SRE example initially rebuilt a search index from retained Shakespeare texts using MapReduce. Rebuilding took little time, so the team did not back up that derived index. The elapsed time mattered alongside derivability: if a comparable rebuild took twelve hours against a thirty-minute target, a snapshot could be a less costly way to meet the target. A rebuild strategy must preserve its sources, code, capacity, and execution order through the failure and measure its duration. When the service later let users add text annotations, the originals could no longer reproduce all valuable data. The team needed to revisit replication for integrity as well as latency and bandwidth, create and test backup/restore procedures, and exercise restoration of annotations within the SLO.

Choose AWS and Microsoft architectures using their applicable workload and service conditions. Google’s historical timing figures illustrate the documented cases; another service needs its own recovery measurements.

These changing reconstruction assumptions imply that a recovery plan is a claim about which data is derivable and which is authoritative, and shipping features moves data across that line without anyone updating the plan. Any feature that lets users create something has probably just invalidated a decision not to back something up. That makes which of our datasets can we rebuild from something else a question worth re-asking on a schedule, not once at design time.

Verify critical data independently of the API that wrote or returned it, because implementation can fail despite a sound interface contract. Out-of-band validators supply another check. Continue investigating plausible failure paths in complex systems rather than assuming current understanding rules them out.

What to do first

Start by recording data dependencies and business requirements, then choose protection and test whether it meets those requirements. Revisit earlier choices when the test or a new feature changes the assumptions.

1. Separate derivable data from authoritative data, and write the list down. Both lists need a recovery plan; only the second necessarily needs backups. For derivable data the plan is the rebuild, so record what it depends on and how long it measured, and drop the backup only where that time fits inside the objective. The list changes when you ship features.
2. Set a data target as well as an availability target, remembering that the same percentage means different things for the two.
3. Classify workloads by business criticality, then assess failure likelihood, impact, and the mechanisms available to contain or recover from each scenario. Decide the HA and DR responsibilities in that architectural context, including rare events.
4. Check that at least one copy is out of reach of whatever could corrupt the original — different credentials, different medium, or immutable for a period. Count failure domains rather than copies.
5. Consider soft deletion where accidental deletion is a material risk. Match its scope and retention to the application’s loss scenarios and deletion obligations; Google’s experience is a reason to investigate, not a universal frequency estimate.
6. Restore something for real, end to end, and time it. Then compare the time to the agreed recovery target, and fix whichever of the two is wrong.

An end-to-end restore converts the plan into measured knowledge, and it is also the one that reveals the gaps nobody predicted — a missing credential, an unavailable decryption key, a schema the current code cannot read, a restore that takes as long as the backup did. All of those are cheap discoveries on a Tuesday and expensive ones during an outage.

The opening team’s replication answer addresses loss of a machine, rack, or region. Recovering from wrong data is a different requirement: replication’s purpose is to make copies agree, and those copies can agree on the same corrupted value.

Check isolation by design and recovery by exercise. Could the thing that corrupts your primary data reach every copy of it — through the same credentials, the same filesystem, the same control plane? And when did you last restore from your oldest layer and measure how long it took? The first is a design question you can answer on a whiteboard today. The second cannot be answered on a whiteboard at all, which is exactly why it is the one that decides whether you recover.

References

• “Data Integrity: What You Read Is What You Wrote,” Google SRE Book
• AWS, Disaster recovery options in the cloud
• Microsoft Learn, Business continuity, high availability, and disaster recovery
• AWS Storage Blog, How to easily replace physical tape-based backups with Tape Gateway