Offline-Online Parity Test

An offline-online parity test checks that the batch code that builds training features and the online code that serves them produce the same values for the same requests at the same point in time. It is a direct test for training-serving skew: instead of waiting for the model to behave strangely, it compares the features themselves.

What is compared

Choose a sample of past requests. For each, the offline path rebuilds the features as they would have been known at the request time, selecting records by when they became available. The online path is replayed over the same event stream, updating its store as events arrive and answering each request from the store’s state at that moment. Both outputs are compared feature by feature and row by row, with a tolerance for floating-point rounding that is much smaller than any meaningful difference. The two inputs to the test are the same events and the same request times; if either differs, a mismatch says nothing about the code.

The example below has one feature, a carrier’s late rate over events that had arrived by request time. The offline function filters stored events with an as-of rule; the online store updates counters incrementally. A second online job is given a unit defect, reporting percent instead of a fraction. Python 3.12.14, standard library only.

import random
random.seed(4)
# events: (arrival_time, carrier, late_flag). The feature: carrier's late rate over events that had ARRIVED by request time.
events = sorted((t + random.uniform(0, 30), "C1", random.random() < 0.2) for t in range(0, 600, 3))
requests = [(t, "C1") for t in range(100, 600, 25)]

def offline(req_t):                 # batch builder: as-of selection on arrival time
    seen = [late for a, c, late in events if a <= req_t]
    return sum(seen) / len(seen)

class OnlineStore:                  # incremental store, updated as events arrive
    def __init__(self, scale=1.0): self.n = self.k = 0; self.scale = scale
    def ingest(self, late): self.n += 1; self.k += late
    def get(self): return self.scale * self.k / self.n

def replay(scale=1.0):
    store, out, i = OnlineStore(scale), [], 0
    for t, _ in requests:
        while i < len(events) and events[i][0] <= t: store.ingest(events[i][2]); i += 1
        out.append(store.get())
    return out

def parity(a, b, tol=1e-9):
    diffs = [abs(x - y) for x, y in zip(a, b)]
    return sum(d > tol for d in diffs), len(diffs), max(diffs)

off = [offline(t) for t, _ in requests]
for name, on in (("as designed", replay()), ("online job reports percent (x100)", replay(100.0))):
    bad, n, worst = parity(off, on)
    print(f"{name:<34} rows differing {bad}/{n}  max abs diff {worst:.4f}  -> {'PASS' if bad == 0 else 'FAIL'}")

as designed                        rows differing 0/20  max abs diff 0.0000  -> PASS
online job reports percent (x100)  rows differing 20/20  max abs diff 30.6000  -> FAIL

The designed job matches on all 20 requests. The defective job fails every row, with a maximum difference of 30.6 on a feature that can only range from 0 to 1. A range check on live traffic would flag the same defect, but the parity test finds it before release and also catches smaller mismatches that stay in range, such as an online store that counts events slightly earlier or later than the batch builder.

What it does and does not cover

• It needs both implementations and a record of what was served. Logging the features at serving time, as the Google guidance recommends, gives a direct comparison with the offline rebuild for real traffic.
• It finds disagreement, not which side is right. Both paths can be wrong in the same way, for example by sharing a mistaken definition, and then they agree. Point-in-time correctness of the definition needs its own tests, such as appending late records and checking that the offline output does not change.
• It tests values at one moment. Freshness, latency, and behavior when the store is stale or missing a key need separate checks.
• It only covers sampled requests. A canary of a few hundred requests catches systematic defects; rare branches may need targeted cases.

Running the test as a release gate, together with sharing code between the paths where possible, reduces the chance that feature store values drift apart from training data. Both are practices the Rules of Machine Learning recommend; the gate covers what code sharing cannot guarantee.

References

• Google, Rules of Machine Learning (Rules 29 and 32, training-serving skew)