Dual Running

Dual running is the practice of running the legacy and the replacement data pipelines side by side on the same inputs for a bounded period and comparing what they produce, before any consumer depends on the new one. It is the data-pipeline form of what the parallel run entry describes for business processes, and the output comparison itself is reconciliation. The legacy outputs stay authoritative throughout, so the new pipeline can be wrong without anyone being harmed, and the point of the exercise is to find out how.

What makes the comparison meaningful

Three conditions decide whether a reported difference is a defect in the new pipeline or noise in the experiment.

• The same frozen input. Each day’s source extract is captured once and both pipelines read that capture. If each reads the live source, rows that arrive between the two reads show up as differences that neither pipeline caused.
• Matching by natural key. Rows are paired by the business identifier, such as order and line number, and not by generated identifiers like surrogate ids or row positions. Generated ids are usually among the things that may differ after a rewrite, and the comparison should be able to report that rather than be broken by it.
• Separate questions. Report rows present in only one output, rows present in both with different values, and aggregates, as three results. Equal row counts and near-equal totals can coexist with thousands of differing rows.

The sketch below compares two implementations of a line-amount calculation that differ only in rounding mode (half up in the old one, half even in the new one), on three days of generated orders. It is a toy that shows what the three results look like, run with Python 3.12.14 on macOS.

def compare(a, b):
    ka, kb = set(a), set(b)
    diff = [k for k in ka & kb if a[k] != b[k]]
    return dict(only_old=len(ka - kb), only_new=len(kb - ka), differing=len(diff),
                total_old=sum(a.values()), total_new=sum(b.values()))

1 {'only_old': 0, 'only_new': 0, 'differing': 36, 'total_old': Decimal('6287.45'), 'total_new': Decimal('6287.09')}
2 {'only_old': 0, 'only_new': 0, 'differing': 22, 'total_old': Decimal('6174.91'), 'total_new': Decimal('6174.69')}
3 {'only_old': 0, 'only_new': 0, 'differing': 26, 'total_old': Decimal('6178.98'), 'total_new': Decimal('6178.72')}

Each day has 400 lines, the key sets are identical, and the totals differ by less than 0.01%. A count check and a loose total check pass. The row-level comparison finds 22 to 36 lines per day that disagree, which is exactly the information the team needs to decide whether the difference is acceptable.

Window length and exit conditions

A dual run can only find differences that its window exercises. Month-end processing, partner batches that arrive every few weeks, late corrections, and holiday calendars occur on their own schedules, so a window of fixed length can pass without ever containing one. A useful exit condition has two parts: an inventory of known source events that the window must have observed, and result criteria, such as no contract violations, no unexplained column differences, and the consumer metrics within the tolerances their owners set. Differences that the team decides to live with are recorded as accepted breaks and excluded from the blocking count; they are not silently ignored.

Dual running also has costs: two platforms’ compute and storage, comparison tooling, and a later finish. It tests that the new pipeline reproduces the old one, so a defect that both share passes. Business invariants and independent references answer that other question and belong beside it. When the window ends, the legacy pipeline normally keeps running for a while as the fallback path, with an owner and a recorded date for switching it off. Moving traffic and keeping a way back is described in the cutover entry. Legacy outputs also contain bugs of their own, so a mismatch does not always mean the new pipeline is wrong: Cartwright, Horn, and Lewis note that new outputs rarely match old ones exactly for this reason.

References

• Ian Cartwright, Rob Horn, and James Lewis: Divert the Flow