Data Platform Disaster Recovery and Cost

In this hypothetical scenario, a faulty release of a transformation job overwrites one column of a revenue table with nulls on Tuesday. The job succeeds, so nothing alerts. A cross-region replica copies the change within minutes, as designed. Two more days of legitimate loads are written on top of the damaged table. On Thursday an analyst notices that revenue per order looks wrong. The team has a nightly backup, a replica in a second region, and a recovery plan on a wiki page. The backup taken on Wednesday night contains the damage. The replica contains the damage. The plan says “restore from backup” and does not say which backup, in what order, or what must exist before the restored files are usable.

This article is about what a data platform needs in order to recover from that kind of event and what it costs to be ready. It follows five topics: the dependencies a restore requires, corruption as a failure mode distinct from loss, recovery drills that turn a plan into evidence, the compute and storage units in which readiness is billed, and the trade-offs between reliability and cost that decide how much protection each dataset deserves. The through-line is that recovery is a property you measure by restoring, and cost is a property you compute from units, so both need numbers and neither can be read off an architecture diagram.

It builds on the previous article, Data SLIs, SLOs, and Incident Response, which defined indicators, objectives, error budgets, and the response to wrong data. Recovery objectives are the same kind of object as an SLO for the worst single event the platform is prepared to survive, and an incident that requires restoring data is the case where detection, scoping, repair, and consumer communication from that article all meet this one. The article on access, retention, and erasure across data copies also matters here for one specific reason: backup retention is simultaneously a recovery parameter and an erasure parameter, and a restore must not undo an erasure.

Replication is not recovery

A platform can lose data in several ways, and the protection that works for one can be useless for another. The table separates five failure kinds and lists what each mechanism does about them. It is a working classification, and real incidents mix kinds.

Failure	Example	Replica in another region	Versioning or time travel	Independent backup
Loss of a location	Region or storage outage	Protects, within replication lag	Lives in the same location, so lost with it unless it is replicated	Protects if stored elsewhere
Accidental delete or overwrite	Dropped table, bad DELETE	Copies the change; depends on whether deletes are replicated	Protects inside the retention window	Protects up to the backup point
Logical corruption	Faulty release writes wrong values	Copies the damage	Protects if the window reaches back past the onset	Protects if a backup predates the onset
Physical corruption	Truncated or damaged files	May copy damaged files	Does not help if the old version’s files are damaged	Protects if the backup is verified and intact
Malicious action	Compromised credential deletes data and backups	Copies deletions it is configured to replicate	Can be deleted with the same credential	Protects if separated by account and credentials

The AWS disaster recovery whitepaper states the central point for object storage in its own words. Continuous replication has the advantage of being the shortest time to back up data, but it may not protect against corruption or malicious deletion as well as point-in-time backups do. For a warm-standby or pilot-light design it adds that continuous data replication protects against some types of disaster but may not protect against data corruption or destruction unless the strategy also includes versioning of stored data or point-in-time recovery options. For multi-site active/active, the most complex and costly strategy, it says recovery time can approach zero for most disasters but that data corruption may need to rely on backups, which usually results in a non-zero recovery point, and that recovery times for a data disaster involving corruption, deletion, or obfuscation will always be greater than zero. The same document recommends separating accounts per Region for isolation in case compromised credentials are part of the scenario. The Google SRE book gives the principle in one sentence: replication and redundancy are not recoverability.

A topology decision (how many places the data lives) answers the loss of a place. It does not answer wrong data, because wrong data is faithfully present in every place. That is why the hypothetical scenario above fails despite having both a replica and a backup: the replica copies the damage, and the question for the backup is which point in time it reaches and whether anyone can find the right one.

Objectives: what recovery has to achieve

NIST’s contingency planning guidance defines the two objectives that frame the rest of this article. The recovery time objective (RTO) is the overall length of time an information system’s components can be in the recovery phase before negatively impacting the organization’s mission or business processes. The recovery point objective (RPO) is the point in time to which data must be recovered after an outage. The RTO and RPO entry stresses that they are separate conditions: a nightly backup that restores a usable service within 30 minutes of detection meets a 45-minute RTO and still fails a five-minute RPO, because the recovered position reflects when the data was captured and not when the restore finished.

Two refinements keep these numbers honest. First, the recovered position is what counts, not the backup schedule. If a snapshot of the 10:00 state finishes copying at 10:40 and is the newest copy when a failure strikes at 11:00, the recovered data is an hour old, because the copy represents the state at its start. Copy frequency, consistent snapshot position, and transfer delay jointly determine exposure. Second, RTO is measured to a usable service, not to restored files. A restore that completes in time but leaves tables unregistered, jobs unscheduled, or credentials missing has not met the objective. Set the objectives per dataset from the consequence of loss or delay, record who accepted them, and treat them as requirements for the recovery design instead of as properties of whatever tooling happens to be installed.

The AWS whitepaper organizes the design space as four strategies with rising cost and falling recovery time: backup and restore, pilot light (data replicated, core infrastructure provisioned but switched off or scaled down), warm standby (a scaled-down but functional copy of the environment), and multi-site active/active. It says backup and restore suits mitigating data loss or corruption, and that for a disaster limited to one data center a highly available workload may need only that. Strategy choice is therefore a statement about the failure kinds in the table above and the objectives per dataset, and it applies to different datasets differently.

Restore dependencies: what must exist before restored data is usable

A backup of data files is not a backup of a working platform. A restore produces a usable service only when everything the data depends on is also present, in a workable order. The inventory below lists the dependencies that data platforms commonly forget, grouped by the thing that breaks.

Dependency	Why the restore fails without it	Platform example
Table metadata	Data files do not say which files belong to which table version	Delta transaction log; Iceberg metadata and manifests
The files the metadata references	A version whose files were removed cannot be read	Files removed by VACUUM, snapshot expiry, or a lifecycle rule
Absolute paths inside metadata	Metadata that points at the old location cannot be used from a new one	Iceberg metadata written before format version 4 uses fully qualified paths
Catalog entries and permissions	Files exist but no query engine knows the table or lets anyone read it	Table registration, grants, row and column policies
Encryption keys and their wrapped copies	Ciphertext without the key is unreadable	Key service, wrapped data keys held with backups
Credentials, networking, accounts	The restore environment cannot reach the backup or the sources	Backup account, role trust, network routes
Infrastructure and code	Nothing to run the restored data on	Infrastructure-as-code definitions, pipeline code, container images
Orchestrator state	Jobs rerun from the wrong position or not at all	DAG definitions, schedules, watermarks, checkpoints, consumer offsets
Replayable upstream	The gap between the backup point and now cannot be refilled	Stream retention long enough to cover the gap; source system extracts
People and procedure	Steps depend on knowledge nobody wrote down	Runbook, access for the people who will execute it

Tables are files plus a log

Lakehouse tables show the first three rows of the inventory clearly. A table in a format such as Delta Lake or Apache Iceberg is a set of immutable data files plus metadata that records which files form each version. Copying the data files alone does not copy the table, and a table cannot be read if the files its metadata references are gone. The script below makes this concrete using the deltalake Python package (delta-rs, version 1.6.6) on local files. It writes three table versions (one append, a second append, then a delete that rewrites the first file), then builds three “backups”: a full directory copy, a copy of only the data files, and a full copy missing the newest data file. All data is invented.

import glob, os, shutil
import pyarrow as pa, pyarrow.parquet as pq
from deltalake import DeltaTable, write_deltalake

shutil.rmtree("work2", ignore_errors=True); os.makedirs("work2")
src = "work2/orders"
def batch(ids): return pa.table({"order_id": ids, "amount": [10 * i for i in ids]})

write_deltalake(src, batch([1, 2, 3]))                 # v0: file A
write_deltalake(src, batch([4, 5]), mode="append")     # v1: file B
DeltaTable(src).delete("order_id = 2")                 # v2: file A is rewritten as A2 (A stays on disk)

def ids(path, **kw): return sorted(DeltaTable(path, **kw).to_pyarrow_table()["order_id"].to_pylist())
def data_files(p): return sorted(os.path.basename(f) for f in glob.glob(p + "/*.parquet"))
print("table read          :", ids(src), "| files on disk:", len(data_files(src)))

# Backup 1: complete directory copy (data files plus _delta_log)
shutil.copytree(src, "work2/backup_full")
# Backup 2: data files only (someone copied "*.parquet")
os.makedirs("work2/backup_files_only")
for f in glob.glob(src + "/*.parquet"): shutil.copy(f, "work2/backup_files_only")
# Backup 3: full copy that is missing the newest data file (interrupted copy or expired by a lifecycle rule)
shutil.copytree(src, "work2/backup_missing_file")
newest = max(glob.glob("work2/backup_missing_file/*.parquet"), key=os.path.getmtime)
os.remove(newest)

print("\nbackup_full         :", ids("work2/backup_full"))
raw = sorted(pq.read_table(f)["order_id"].to_pylist() for f in glob.glob("work2/backup_files_only/*.parquet"))
print("files_only, union   :", sorted(x for r in raw for x in r), "(no log: cannot tell which files are current)")
try:
    DeltaTable("work2/backup_files_only")
except Exception as e:
    print("files_only as table :", type(e).__name__)
try:
    print("missing_file latest :", ids("work2/backup_missing_file"))
except Exception as e:
    print("missing_file latest :", type(e).__name__)
print("missing_file, v1    :", ids("work2/backup_missing_file", version=1))

table read          : [1, 3, 4, 5] | files on disk: 3

backup_full         : [1, 3, 4, 5]
files_only, union   : [1, 1, 2, 3, 3, 4, 5] (no log: cannot tell which files are current)
files_only as table : TableNotFoundError
missing_file latest : FileNotFoundError
missing_file, v1    : [1, 2, 3, 4, 5]

Reading the output. The table, with its log, reads as orders 1, 3, 4, and 5 (order 2 was deleted), and three data files are on disk because the delete rewrote the first file and left the old one in place. The full copy reproduces the table exactly. The data-files-only copy cannot be opened as a table (TableNotFoundError), and reading its files directly returns every row of every file, including order 2 and duplicates of orders 1 and 3, because without the log nothing says which files are current. The third copy is missing one data file, so the latest version cannot be read, yet version 1 can still be read, because its files are all present. That last line shows a useful property and a trap. Table history gives more than one place to land, but which versions are restorable depends on which files survived, so the question “can we restore” has to be asked of a specific version.

Delta Lake’s documentation states the same constraint for its RESTORE command: restoring to an older version whose data files were deleted manually or by VACUUM will fail, and partial restoration is possible only with spark.sql.files.ignoreMissingFiles set to true. Its VACUUM documentation says that the ability to time travel to a version older than the retention period is lost after running vacuum. Apache Iceberg’s expiry procedure similarly removes snapshots, and the files exclusively referenced by them, according to age settings. Retention settings on a table therefore define the furthest restore point inside the table itself, independent of any external backup.

Iceberg adds a portability dependency. Its specification states that before format version 4, all path fields in metadata must contain fully qualified paths, and path strings that start with a scheme such as s3: or gs: are used as-is. A copy of the files at a new location carries metadata that still points at the old one. Iceberg’s Spark procedures include rewrite_table_path, which stages a copy of the table’s metadata with a source prefix replaced by a target prefix; the documentation says it does not copy the data files itself. Restoring an Iceberg table to a different location is thus a two-part operation, and a plan that says “copy the bucket” omits the second part. Registering the restored table in a catalog is a further step: the register_table procedure creates a catalog entry for a metadata file that already exists but has no catalog identifier.

Keys, grants, orchestrator state, and replay

The other dependencies fail differently. Encrypted data needs its keys, and the interaction with erasure is a trap. If an erasure was implemented by destroying a per-person key, then restoring the key together with the ciphertext restores the person’s data. Equally, if backups were encrypted under a key that was later rotated out or deleted without a record of which backups depend on it, those backups are unreadable. The envelope encryption entry makes the inventory point: include wrapped data keys in backups and the inventory, and check every dependent copy before retiring a key version.

Orchestrator state is the dependency most often missing from the plan. A restored table is a state at a point in time. The pipelines that keep it current need to know what that point is: a watermark, a stream checkpoint, a consumer offset, or a completed-partition marker. If the orchestrator metadata is restored to a different moment than the data, jobs either reprocess data that is already there (harmless if the pipeline is idempotent, a duplication source if not) or skip a range (silent data loss). Replay then fills the gap between the backup point and the present, and it depends on the upstream still holding that range. A stream retained for three days cannot refill a gap of five.

Dependencies have an order, and the order determines the recovery time. The calculation below models a hypothetical restore plan as a dependency graph with assumed step durations and finds the critical path. The restore time for the data step is computed from an assumed volume and sustained throughput. None of the numbers is a measurement; the point is the structure.

# Illustrative restore plan. Durations are assumptions, not measurements.
DATA_TB, MBPS = 12, 800                      # data to restore and sustained restore throughput
restore_min = DATA_TB * 1_000_000 / MBPS / 60
steps = {                                    # name: (minutes, [prerequisites])
  "declare and decide":              (30, []),
  "provision storage and network":   (40, ["declare and decide"]),
  "restore keys and credentials":    (20, ["provision storage and network"]),
  "restore orchestrator metadata":   (20, ["provision storage and network"]),
  "restore table data":              (restore_min, ["restore keys and credentials"]),
  "register tables, grants":         (25, ["restore table data"]),
  "replay stream from checkpoint":   (120, ["register tables, grants", "restore orchestrator metadata"]),
  "validate by reconciliation":      (45, ["replay stream from checkpoint"]),
  "reopen consumers, notify":        (15, ["validate by reconciliation"]),
}
finish, via = {}, {}
def done(n):
    if n in finish: return finish[n]
    dur, pre = steps[n]
    start = max((done(p) for p in pre), default=0)
    via[n] = max(pre, key=done) if pre else None
    finish[n] = start + dur
    return finish[n]
end = max(steps, key=done)
path, n = [], end
while n: path.append(n); n = via[n]
print(f"restore table data: {restore_min:.0f} min ({DATA_TB} TB at {MBPS} MB/s)")
print("critical path:", " -> ".join(reversed(path)))
print(f"estimated recovery time: {finish[end]:.0f} min = {finish[end]/60:.1f} h")
for p, (d, _) in steps.items():
    print(f"  {p:32s} {d:6.0f} min  finishes at {finish[p]:6.0f}")

restore table data: 250 min (12 TB at 800 MB/s)
critical path: declare and decide -> provision storage and network -> restore keys and credentials -> restore table data -> register tables, grants -> replay stream from checkpoint -> validate by reconciliation -> reopen consumers, notify
estimated recovery time: 545 min = 9.1 h
  declare and decide                   30 min  finishes at     30
  provision storage and network        40 min  finishes at     70
  restore keys and credentials         20 min  finishes at     90
  restore orchestrator metadata        20 min  finishes at     90
  restore table data                  250 min  finishes at    340
  register tables, grants              25 min  finishes at    365
  replay stream from checkpoint       120 min  finishes at    485
  validate by reconciliation           45 min  finishes at    530
  reopen consumers, notify             15 min  finishes at    545

Reading the output. Under these assumptions the estimated recovery time is 545 minutes, about nine hours, and the critical path runs through provisioning, key and credential restoration, the 250-minute data restore, registration, replay, and validation. Two things stand out. The orchestrator-metadata step is off the critical path because it finishes long before replay can begin, so speeding it up changes nothing. The data transfer dominates, so its throughput assumption is where an RTO estimate is most sensitive: halving the throughput to 400 MB/s would add 250 minutes. And the steps other than the data restore itself (deciding, provisioning, keys, registration, replay, validation, reopening) account for 295 of the 545 minutes. A plan that quotes only “time to copy the data” would understate the recovery time by more than half under these assumptions.

A public case: restores meet dependencies

The Google SRE book’s data integrity chapter describes a real incident that shows several of these dependencies at once. A refactored data-deletion pipeline for Google Music, built to honor user deletion requests, removed about 600,000 audio references that it should not have, affecting audio files for 21,000 users, and the damage was found more than a month after the pipeline’s first run. Recovery used offsite tape backups: the initial recovery was split into 5,475 restore jobs that engineers submitted programmatically, since typing them would have taken a human operator more than three days; 5,337 tapes were recalled by truck; a vendor omitted 1,862 tapes from the first recall; and 17 tapes were bad, which a redundant encoding was designed to tolerate. The 1.5-petabyte recovery took longer than the two days estimated, and the account says the restore of the first batch finished in just under seven days from the start, five of them for the data recovery itself. About 161,000 tracks had been deleted before they were ever backed up; most were store-bought or promotional tracks whose original copies were unaffected, and user-uploaded files were recovered by asking clients to re-upload.

The source states those facts; the interpretation that follows is this article’s. The case illustrates a restore that depended on a physical inventory (which tapes), a recall procedure, a mapping from files to backups, and a decision about data that was never backed up. It also shows an estimate coming from rehearsal and still being exceeded, which is the normal relationship between a drill and a real event. And it connects this article to the previous one on erasure: the incident was caused by a deletion pipeline that had to remove data for privacy reasons, which is a reminder that erasure jobs are destructive at scale and need the same blast-radius controls and monitoring as any other deletion path.

Corruption: wrong data is the failure replication cannot fix

Corruption differs from loss because the data is still there and still looks like data. Two forms matter. Logical corruption is data that is well-formed and wrong: a faulty transformation, a bad backfill, a unit change. Physical corruption is damage to the bytes: a truncated file, a partially written object, a bad disk sector. Logical corruption is usually the more common and the harder one, because every consistency check that looks at structure passes.

Three properties make corruption expensive. It is detected late, since the symptom is a wrong number noticed by a consumer. The SRE book notes that low-grade mutation or deletion bugs in application code demand the furthest reaches back in time, as some were noticed months after the first loss began. It spreads, because each downstream job copies and transforms the bad values, and each replica copies them. And recovery has a decision at its center that loss does not: to which moment do you go back, and what do you do about the legitimate work done since?

Finding the onset

Restoring requires a target. The point-in-time recovery entry names three ways to find a defensible one, and they apply to table versions as much as to database logs. Work from a change: if the corruption began with a deployment, a migration, or a configuration change, its timestamp is the candidate. Work from validation: the last passing and first failing checks bound the onset, but only if those checks would reliably have detected the fault throughout the period. Search: restore candidate moments in isolation, run a check, and move the candidate. The search is expensive when each probe is a full restore, and cheap when the table format keeps versions queryable in place.

The script below uses that last property. It builds a six-version Delta table in which version 3 is a faulty release that nulls four amounts, versions 4 and 5 are legitimate appends, and it then finds the first bad version by bisection, reading each probed version through time travel. It compares two recovery options: restore the whole table to the last good version, or repair forward by merging the good values back into the current table. It was executed locally with the same package version as above; the data is invented.

import os, shutil
import pyarrow as pa
from deltalake import DeltaTable, write_deltalake

shutil.rmtree("work3", ignore_errors=True); os.makedirs("work3")
p = "work3/orders"
def batch(ids): return pa.table({"order_id": pa.array(ids, pa.int64()), "amount": pa.array([10 * i for i in ids], pa.int64())})

write_deltalake(p, batch(range(1, 7)))                              # v0
write_deltalake(p, batch([7, 8]), mode="append")                    # v1
write_deltalake(p, batch([9, 10]), mode="append")                   # v2
DeltaTable(p).update({"amount": "NULL"}, predicate="order_id IN (3,4,5,6)")   # v3: faulty release nulls four amounts
write_deltalake(p, batch([11, 12]), mode="append")                  # v4: legitimate
write_deltalake(p, batch([13, 14]), mode="append")                  # v5: legitimate

def bad_rows(version):
    t = DeltaTable(p, version=version).to_pyarrow_table()
    return sum(1 for v in t["amount"].to_pylist() if v is None)

latest = DeltaTable(p).version()
print("latest version:", latest, "| rows with NULL amount now:", bad_rows(latest))

lo, hi = 0, latest                     # invariant: version lo is good, version hi is bad
assert bad_rows(lo) == 0 and bad_rows(hi) > 0
checks = 0
while hi - lo > 1:
    mid = (lo + hi) // 2; checks += 1
    if bad_rows(mid) == 0: lo = mid
    else: hi = mid
print("last good version:", lo, "| first bad version:", hi, "| versions inspected:", checks + 2)

# Option A: restore the whole table to the last good version
a = "work3/orders_restore"; shutil.copytree(p, a)
DeltaTable(a).restore(lo)
ta = DeltaTable(a).to_pyarrow_table()
print("A restore  -> rows:", ta.num_rows, "| NULL amounts:", sum(v is None for v in ta["amount"].to_pylist()), "| lost legitimate orders:", [i for i in (11, 12, 13, 14) if i not in ta["order_id"].to_pylist()])

# Option B: repair forward, copying the good values from the last good version and keeping later work
b = "work3/orders_repair"; shutil.copytree(p, b)
good = DeltaTable(p, version=lo).to_pyarrow_table()
DeltaTable(b).merge(source=good, predicate="t.order_id = s.order_id AND t.amount IS NULL",
                    source_alias="s", target_alias="t").when_matched_update({"amount": "s.amount"}).execute()
tb = DeltaTable(b).to_pyarrow_table()
print("B repair   -> rows:", tb.num_rows, "| NULL amounts:", sum(v is None for v in tb["amount"].to_pylist()), "| lost legitimate orders:", [i for i in (11, 12, 13, 14) if i not in tb["order_id"].to_pylist()])

latest version: 5 | rows with NULL amount now: 4
last good version: 2 | first bad version: 3 | versions inspected: 4
A restore  -> rows: 10 | NULL amounts: 0 | lost legitimate orders: [11, 12, 13, 14]
B repair   -> rows: 14 | NULL amounts: 0 | lost legitimate orders: []

Reading the output. At version 5 the table has four null amounts. Bisection needed to read four versions (the two endpoints plus two probes) to establish that version 2 is the last good one and version 3 the first bad one, instead of scanning all six. With many versions the saving grows logarithmically, which is why queryable history makes the search affordable. The two options then differ in what they lose. Restoring to version 2 produces ten clean rows and discards four legitimate orders (11 to 14) that were written after the damage. Repairing forward keeps all 14 rows and leaves no nulls, because it copies the four good amounts from version 2 into the matching null rows in the current table. The comparison does not make repair universally better. Repair forward works here because the damage is confined to a known column on known keys and the good values still exist. If the corruption were pervasive, or the good values had been overwritten and vacuumed, restoration plus reprocessing the later work from raw inputs would be the available route.

This is the same trade-off the forward recovery entry describes for cutovers, in a data setting: returning to a prior state discards valid work done since, and some effects (messages sent, reports published, decisions made) cannot be reversed by restoring data. After any restore, the incident practice from the previous article applies in full: announce what changed, restate affected figures, and reconcile.

Detecting and bounding corruption

Physical corruption is addressed by checksums and verification, which compare stored bytes with a reference. A chunked checksum compares summaries of bounded key ranges, so a large copy can be verified without transferring every row, with the caveat that a matching hash is evidence and not proof, and that both sides need the same definition of row order, nulls, and representation. Logical corruption needs the checks from the previous article: reconciliation against an independent reference, invariants on values, and distribution monitoring. A restore that is not followed by reconciliation shows that files came back, not that the data is right.

Containment is the other half. The blast radius of a bad release is set by what it can write to and how gradually it is applied. Writing new versions to a staging location and validating before an atomic switch, running a change on one dataset or partition first, and retaining enough table history to reach back past the validation interval all bound the damage. Retention of history is itself a cost, which the cost sections below quantify.

Recovery drills: turning a plan into a measured fact

A recovery plan that has not been exercised is a hypothesis about a system that has since changed. A recovery drill is a deliberate rehearsal of a restore or failover when nothing is wrong, to establish that it works and how long it takes. The SRE book’s heading for the principle is “Hope Is Not a Strategy,” and its sharper statement is that you only know you can recover your recent state if you actually do so. A successful recovery last quarter does not exempt this quarter, because parts of the process can break in between.

A drill needs pass criteria decided before it starts, or the result is “we restored something.” The criteria below are this article’s arrangement of what the sources require.

1. Inside the recovery time objective. Define the clock’s start and the usable-service condition in advance. Include retrieval, provisioning, replay, validation, routing, and access checks at representative volume. A restore of a small sample proves a narrower capability than a full-volume restore.
2. Inside the recovery point objective. Record the recovered position and the accepted work missing from it, and compare with the objective. A backup can open quickly and pass structural checks while missing a day of orders; a completeness or freshness check against a reference detects that.
3. The data is right. Check correctness with validators independent of the restore mechanism (the SRE book advises out-of-band data validators even when API semantics suggest they are unnecessary), including reconciliation against a source.
4. The procedure carried it, not a person. If the drill succeeded because one engineer remembers how the system works, the test measured that engineer’s availability.
5. Everything needed was reachable. Credentials, keys, the runbook itself, and the account that holds the backups, under the conditions of the scenario and not under normal conditions.
6. Return to service was controlled. Name who can stop the drill, the limits that trigger a stop, and the path back. An isolated restore needs cleanup and a guard against sending real notifications.

Report each objective as passed, failed, or untested, separately from what the drill taught. Finding a broken step is useful learning and does not turn a missed RTO into a pass. The gaps a first drill typically finds are mundane: an expired credential, a key held by someone who left, a backup whose schema the current code cannot read, a restore that takes as long as the backup did because nobody measured it, and a dependency order that nobody wrote down. The critical-path calculation above is a way to write the order down and to find which step to measure first.

Scale drills up in steps the team can observe and stop. Read the procedure aloud and fix steps that name systems that no longer exist. Walk a specific scenario through the plan with the people who would be involved and decide, not recite. Restore for real into an isolated environment and time it. Exercise the failover itself where one exists. The AWS whitepaper adds that for a multi-site active/active design there is no failover to invoke, so the test focuses on loss of a Region and on whether the remaining Regions can carry the traffic, and that testing for a data disaster is separately required. A topology test and a data test check different failures.

Cadence follows from what decays. Systems change continuously, so the interval should be short enough that nothing important has changed since the last drill. Automating the frequent part (a nightly restore of a sample with automated validation) and running a full human drill on a slower schedule is an arrangement that tends to survive competing priorities, though the SRE source warns that automation staffed by engineers with competing priorities can produce temporary stopgaps. Drills also cost money, which belongs in the cost model below: an isolated restore uses storage, compute, and sometimes transfer, and its cost is part of the price of knowing.

Compute and storage units: what readiness is billed in

Cost engineering starts by naming the unit. A bill is a sum of quantities, each measured in a unit the provider defines, multiplied by a price. Prices change and differ by contract, so engineering decisions are best expressed in units, with prices applied last. For a data platform readiness appears in four families of units: bytes stored over time, bytes moved, compute used over time, and operations. The table lists the units used by the example systems in this article.

Family	Unit	How it accrues	Design lever
Storage	Bytes stored, averaged over time (for example GB-months)	Continuously, including old versions, time travel and fail-safe data, and backups	Retention windows; churn; compaction; storage class; compression
Storage	Minimum storage duration and minimum object size on some classes	Removing or changing data early leaves a charge for the remaining period	Choose classes by how long data will actually live; check transition rules
Transfer	Bytes moved across a billed boundary	Replication, restore, cross-region reads, egress	Where copies live; incremental versus full refresh; compress or filter before moving
Compute	Snowflake credit: warehouse size per hour, billed per second with a 60-second minimum each start	While a warehouse runs; each size up roughly doubles the rate (documented: Small 2 credits per hour, Medium 4)	Size, auto-suspend, concurrency by more warehouses or clusters, not size
Compute	BigQuery slot-hour under capacity pricing, or bytes processed per TiB under on-demand	Reserved baseline always billed; autoscale billed as used; on-demand billed per bytes scanned	Baseline versus autoscale; scan volume through layout and pruning
Compute	Capacity unit (Microsoft Fabric), a pool of compute shared by all workloads on a capacity	As long as the capacity is provisioned, at a rate set by its size	Separate capacities for workloads that must not compete; size to combined peak
Operations	Requests, retrievals, restores	Per request or per retrieved byte, depending on the storage class and retrieval mode	Fewer, larger objects; avoid restoring through expensive tiers in drills

The Snowflake credit and the virtual warehouse entries make one point that matters for recovery cost: credits measure compute consumption, and how many a workload uses is an engineering question while what each credit costs is commercial. A Medium warehouse (4 credits per hour) running two hours consumes 8 credits; the same work on a Large warehouse (8 credits per hour) consumes 8 credits only if the job really finishes twice as fast, which is a condition to test. The one-minute minimum means repeated start-and-stop within the first minute is billed repeatedly. In BigQuery, the slot is an abstracted unit that bundles compute, memory, and I/O, and under capacity pricing the baseline of a reservation is charged whether or not anyone queries, while autoscale slots appear only under load. Fabric’s capacity unit is a pool shared by every workload on the capacity, which makes it both a budget and a point of contention. In every case the unit determines which lever works: sizing and suspension for warehouses, baseline versus autoscale for slots, separation of workloads for shared capacity.

History and recovery storage multiply bytes

Recovery features are mostly storage. Every retained version, time-travel window, fail-safe period, and backup is bytes stored for a period. The documentation read for this article gives specifics. Snowflake’s Time Travel defaults to 1 day and can be set up to 90 days for permanent objects on Enterprise Edition and above, followed by a non-configurable 7-day Fail-safe that the vendor operates on a best-effort basis and that its documentation says is not a way to access historical data after Time Travel ends; its recovery may take hours to days and requires a support case. BigQuery’s time travel window is 7 days by default, configurable from 2 to 7, followed by a non-configurable 7-day fail-safe from which data is recoverable at table level only through Cloud Customer Care. For billing, BigQuery’s documentation says that under the logical storage billing model time travel and fail-safe storage are included in the base rate, while under the physical model they are billed at the active physical rate. Delta Lake’s defaults (7 days for removed files, 30 days for log history) and Iceberg’s snapshot expiry settings are properties of the table and consume storage in the buckets the owner pays for.

The hypothetical calculation below estimates retained-history storage for a 100 TB table in which 3 percent of the bytes change per day. The unit prices are invented and are not any provider’s list prices. The model assumes retained history grows with the bytes changed over the retention period, which is a simplification that overstates cases where changes overwrite the same bytes and understates cases where compaction rewrites more data.

# Hypothetical unit prices, NOT any provider's price list. The point is which unit dominates.
P_STORE   = 20      # per TB-month stored
P_XFER    = 80      # per TB transferred across regions
P_CREDIT  = 3       # per compute credit (one credit = one unit of compute for one hour)
DATA_TB, CHURN, HOURS = 100, 0.03, 720   # primary size, fraction of bytes changed per day, hours per month
PRIMARY_CREDITS_PER_HOUR = 40            # average compute the primary workload uses

def money(x): return f"{x:,.0f}"
rows = []
# A: in-place history only (versions / time travel), no second copy
hist_tb = DATA_TB * CHURN * 14
rows.append(("A history only (14 days)", hist_tb * P_STORE, 0, 0, 0))
# B: cross-region backup copy, 35-day retention of changes, restore on demand
daily_tb = DATA_TB * CHURN
b_store = (DATA_TB + daily_tb * 35) * P_STORE
b_xfer = daily_tb * 30 * P_XFER
rows.append(("B backup and restore", b_store, b_xfer, 0, 0))
# C: warm standby: replicated copy plus a scaled-down always-on compute (2 credits/hour)
c_store = DATA_TB * P_STORE + hist_tb * P_STORE
rows.append(("C warm standby", c_store, daily_tb * 30 * P_XFER, 2 * HOURS * P_CREDIT, 0))
# D: active/active: second full compute fleet
rows.append(("D active/active", c_store, daily_tb * 30 * P_XFER, PRIMARY_CREDITS_PER_HOUR * HOURS * P_CREDIT, 0))
print(f"{'strategy':28s}{'storage':>10s}{'transfer':>10s}{'compute':>10s}{'total/month':>13s}")
for n, s, x, c, _ in rows:
    print(f"{n:28s}{money(s):>10s}{money(x):>10s}{money(c):>10s}{money(s+x+c):>13s}")
print()
print("retained-history storage for a 100 TB table with 3% daily change:")
for tt, fs in ((1, 7), (7, 7), (30, 7), (90, 7)):
    tb = DATA_TB * CHURN * (tt + fs)
    print(f"  time travel {tt:2d} d + fail-safe {fs} d -> {tb:6.1f} TB retained -> {money(tb*P_STORE)} per month")

strategy                       storage  transfer   compute  total/month
A history only (14 days)           840         0         0          840
B backup and restore             4,100     7,200         0       11,300
C warm standby                   2,840     7,200     4,320       14,360
D active/active                  2,840     7,200    86,400       96,440

retained-history storage for a 100 TB table with 3% daily change:
  time travel  1 d + fail-safe 7 d ->   24.0 TB retained -> 480 per month
  time travel  7 d + fail-safe 7 d ->   42.0 TB retained -> 840 per month
  time travel 30 d + fail-safe 7 d ->  111.0 TB retained -> 2,220 per month
  time travel 90 d + fail-safe 7 d ->  291.0 TB retained -> 5,820 per month

Reading the output. The second block isolates one decision: the retention window. Moving from 1 day of time travel to 90 days, with a fixed 7-day fail-safe, raises retained history from 24 TB to 291 TB under the stated churn, and under the assumed price from 480 to 5,820 per month, a twelvefold increase for a ninety-fold longer window (the fail-safe is a fixed term in both). That is the price of the ability to reach 90 days back, and it needs to be set against the value of reaching that far, which the corruption section argues can be months for low-grade bugs. Under BigQuery’s default logical billing model that line would not appear as a separate charge, which is a reminder that the same recovery feature maps to different bills in different products.

The first block compares four strategy shapes under the same assumed prices and a primary workload that uses 40 credits per hour on average. Each strategy is dominated by a different unit. History-only protection (A) is a small storage line and gives no protection against loss of the location. Backup and restore (B) is dominated by transfer to the second location, 7,200 of 11,300 per month, because 3 TB changes daily and must cross a billed boundary (the model omits the one-time seeding of 100 TB, which at the assumed price would add 8,000 once). Warm standby (C) adds a small always-on compute line and a replica. Active/active (D) is dominated by compute, 86,400 of 96,440, because it runs a second full fleet. The totals are not predictions; they show where to look. If transfer dominates, reducing what crosses the boundary (compressing, replicating incrementally, filtering) is the lever. If compute dominates, the lever is how much standby capacity you really need.

Restore itself has costs not visible in the monthly totals. The archive rehydration entry warns that a restore request is not a completed recovery, and that waiting, downloading, decrypting, and rebuilding downstream data belong in the deadline; retrieval and copy charges belong in the cost. A storage lifecycle policy that moves backups to colder storage lowers the storage line while raising restore time and retrieval charges, and the minimum storage duration of the cold class means deleting early still charges for the remainder. Data transfer cost is keyed to the exact path, direction, and contract, so topology identifies which prices to check but does not rank boundaries by itself, and replication and recovery flows must be included in the inventory.

Reliability and cost: deciding how much protection each dataset deserves

Both reliability and cost are unbounded: there is always another copy, another drill, another standby. The decision is where to stop for each dataset, and it is a comparison of two quantities. One is the expected cost of the failure the protection addresses, in business terms: revenue delayed, decisions made on wrong data, contractual penalties, regulatory exposure. The other is the cost of the capability, in the units above. Neither side is precise. The value of the approach is that it forces both to be written down per dataset instead of applying one policy to the whole platform.

A practical structure is tiers defined by consequence. The table is illustrative: the tier names, objectives, and strategies are examples of the form, not recommendations for a particular organization.

Tier (illustrative)	Example datasets	RTO and RPO (assumed)	Protection shape	Drill cadence
Tier 0: revenue and compliance critical	Billing, financial close, regulated records	Hours; minutes of loss or less	Cross-account backup with point-in-time reach; warm standby or better for serving paths; separate credentials	Quarterly full; frequent automated sample
Tier 1: operational analytics	Core dimensional models, daily reporting	Same day; up to a day of loss	Table history plus independent backup; rebuild from raw as a documented option	Twice a year full; automated sample monthly
Tier 2: derived and rebuildable	Aggregates, marts reproducible from raw	Days; rebuild from raw	Protect the raw inputs and the code; derived tables need only enough history for investigation	Rebuild drill yearly
Tier 3: scratch	Sandboxes, temporary tables	None	Short retention; no backup	None

The tiers contain a design idea worth stating directly. For rebuildable data, the protection is the raw layer, the code, and the orchestrator state, and recovery means recomputation. Rebuilding is a compute cost and a time cost, and it is only available if the transformations are idempotent and the raw data is retained long enough. Comparing “restore” and “rebuild” for a dataset means comparing a storage-and-transfer bill, paid continuously, with a compute bill, paid only in an incident, plus the time each takes. For the critical path example above, a restore of 12 TB at 800 MB/s takes 250 minutes; a rebuild of the same data would be limited by raw volume and cluster size, which only a measurement can answer.

Several cost controls interact with reliability, and each has a documented limit that should be known before relying on it. As the resource monitor entry summarizes Snowflake’s documentation, a monitor tracks credits used by warehouses and can notify or suspend them at quota thresholds, but it does not track serverless features, storage, or data transfer, and it is not an exact cap because warehouses take time to suspend. An immediate suspension cancels running statements, so a multi-step pipeline can be left half done unless its steps can be safely re-run, which makes idempotent pipeline design a prerequisite for aggressive cost guardrails. A commitment discount trades a lower rate for an obligation, and the commitment is paid whether or not usage materializes; break-even utilization is the discounted price over the on-demand price (in the glossary’s hypothetical, a commitment priced at 70 against 100 on demand breaks even at 70 percent utilization). Standby capacity in a warm-standby design is a poor candidate for such commitments because its utilization is low by design.

Visibility is part of the control. A cost allocation tag groups costs by team, product, or environment, but only if something applies it consistently and billing is configured to use it, and some costs (transfer, support) are not resource-shaped. Showback reports recovery and storage costs to the teams whose designs create them without charging their budgets (chargeback does charge them), and unit economics relates cost to a defined unit of output, such as cost per terabyte protected or per tested restore. A unit cost rising does not by itself show waste; activity, quality, and requirements can change, so read trends with totals and context. The protective measure to track beside cost is the drill record: the cost of readiness is only justified if the recovery objectives were actually demonstrated.

Where cost-cutting quietly removes recovery

• Shortening retention to save storage. A shorter time-travel or snapshot window moves the furthest restore point closer. If low-grade corruption is noticed after the window, the data is gone. Set the window from how long detection actually takes, and measure that from incident history.
• Moving backups to cold storage without counting retrieval. The storage line falls and the RTO rises. Include retrieval time and charges in the drill.
• Deleting what looks orphaned. Iceberg’s orphan file removal and Delta’s VACUUM delete files nothing references; a backup scheme that depends on files the table no longer references will find them gone. Check what restores depend on before cleanup jobs run.
• Dropping drills as a cost. The drill is how objectives become facts. Reduce its cost with isolated, sampled, automated restores instead of removing it.
• Treating replication as the backup. It is cheaper than point-in-time protection and does not address corruption or malicious deletion, as the earlier table shows.
• Backups that outlive erasure promises. Long backup retention lengthens the period in which erased data persists, as the previous article discusses. Retention here is a privacy parameter, so set it with that obligation and apply the erasure list on restore.

Failure modes and limits

• The plan names the data and omits the dependencies. Keys, catalog entries, orchestrator state, and replay sources are part of the restore. Write the dependency graph down and find its critical path.
• Objectives with no evidence. An RTO or RPO that has never been exercised is a target and not a result. Report passed, failed, or untested.
• One backup, one point. If the only backup postdates the corruption, restoration cannot help. Retain history long enough to cover detection delay, and keep history independent of the credentials that can corrupt the primary.
• Restore without reconciliation. Files that come back are not data that is right. Reconcile against an independent reference.
• Estimates from the best case. The public case above exceeded a two-day estimate despite regular drills. Use ranges, state assumptions, and revise from measured drills.
• Cost models that ignore the dominant unit. Find which unit dominates each strategy before optimizing the wrong line.
• Prices treated as facts. Provider prices, defaults, and limits change. The windows and defaults cited here were checked against documentation in early October 2026 and should be rechecked when a decision depends on them.

None of this removes the need for judgment, because the inputs (value of data, tolerance for delay, likelihood of corruption) are estimates. What the structure provides is an explicit statement of those estimates per dataset, a restore plan whose dependencies and critical path are written down, evidence from drills instead of belief, and a cost model expressed in units so that decisions survive price changes. Recovery that has been rehearsed and costed is a capability the organization can state accurately, including its limits, and that is what recovery objectives, error budgets, and cost reviews all depend on.

References

• AWS Whitepaper, Disaster Recovery of Workloads on AWS: Disaster recovery options in the cloud
• Google, Site Reliability Engineering: Data Integrity: What You Read Is What You Wrote (includes the Google Music case study)
• NIST CSRC Glossary, recovery time objective (source: NIST SP 800-34 Rev. 1)
• NIST CSRC Glossary, recovery point objective (source: NIST SP 800-34 Rev. 1)
• Delta Lake documentation, Table utility commands (VACUUM and RESTORE)
• Delta Lake documentation, Table batch reads and writes (time travel and data retention)
• Apache Iceberg documentation, Maintenance
• Apache Iceberg documentation, Spark Procedures (expire_snapshots, rewrite_table_path, register_table)
• Apache Iceberg Table Spec (path resolution)
• Snowflake documentation, Understanding and using Time Travel
• Snowflake documentation, Understanding and viewing Fail-safe
• Snowflake documentation, Understanding compute cost
• Snowflake documentation, Overview of warehouses
• Google Cloud, BigQuery data retention with time travel and fail-safe
• Google Cloud, BigQuery pricing (on-demand and capacity compute models, storage)
• Google Cloud, BigQuery slots (reservations, baseline, and autoscaling)