Forward Recovery

Forward recovery restores a usable state by keeping the new system authoritative and repairing it in place with controlled, reviewed corrections, instead of returning to the previous system. Rollback and forward recovery are the two ways out of a bad cutover; which one is available depends on what has happened since the switch.

Why rollback stops being an option

Suppose a new order-entry system has accepted orders, updates, and deletions for a day. Returning users to the old database without carrying that work back discards valid business activity. Some effects cannot be reversed by restoring data at all: confirmation emails already sent, approvals already granted, payment requests already issued, identifiers already used by external partners. Rollback returns a system to a prior version; it does not unsend a message.

Decide the point of no return before the cutover, for example once external partners have consumed identifiers the new system issued, and state which recovery applies on each side of it. Before that point, a tested synchronization can carry new work back and rollback remains possible. After it, forward recovery is the plan.

Repair in place, under control

Forward recovery means identifying the faulty records or behavior, correcting them with reviewed changes, and verifying the result against the agreed requirements. Keeping the new system authoritative does not mean keeping it fully open: depending on the damage, writes may be paused, the service set read-only, or particular functions such as payment requests suspended until the correction is verified, and then resumed. Forward recovery is not the same as uninterrupted recovery. It needs the same discipline as any change: an owner, a record of what was corrected and why, and reconciliation afterwards. Google’s reliability engineering guidance stresses that the sooner a data problem is detected, the easier and more complete recovery can be, because bad data propagates as it is copied and updated.

For a read-only reporting path the question rarely arises, because nothing new was written; pointing users back to the old report is enough provided the old job still runs with valid access, its data is current enough, its output is still trusted, and notifications already sent by the new report are dealt with. The harder questions belong to systems that accept transactions, and they should be answered before the first one is accepted.

Reference: Google SRE: Data integrity. Examples here are illustrative.