Case: Data Platform Migration for AI Consumers

Halden Parts is a fictional distributor of industrial replacement parts. The company, its systems, and every number below were constructed for this case; it does not describe a real migration. The measurements come from a small simulation, described as it is used, that runs on synthetic data.

Halden’s analytics rest on a warehouse that a nightly batch job has rebuilt for years. A new platform is ready to replace it. The migration team has rewritten three datasets as new pipelines: a product dimension, an order-line fact table, and per-customer features. Two systems read those datasets, and neither belongs to the migration team. A product search service turns each product’s text into a vector and ranks results by similarity. A weekly reminder campaign scores every customer with a reorder-propensity model whose inputs are the customer features. Neither consumer sends a query that a migration test suite would notice. Each consumes what the pipelines produce, and a rewritten pipeline can produce output with the same columns and the same row counts that means something different.

The question is how the team shows, before it moves either consumer, that the new platform preserves what each one depends on, and how it moves them in a way that can be undone. The design that follows combines data contracts, dual running, reconciliation at several depths, tolerances owned by the consumers, a decision rule written as code, and a cutover that proceeds one consumer at a time. After reading it you should be able to explain that design, reproduce its tests on your own datasets, and say what the tests cannot see.

The case assumes the earlier articles on Designing Topics and Event Contracts, Data Pipeline CI/CD and Environment Promotion (contract tests and expand-and-contract migration), Data Quality Invariants and Business Reconciliation, Metadata and Lineage for Change Impact Analysis, Feature Pipelines and Point-in-Time Correctness, Document Pipelines for Retrieval, and Building Incremental and Idempotent Pipelines. The previous case, Case: A Predictive Service with Reliable Features and Recovery, builds a feature-backed prediction service with evaluation, monitoring, and rollback. Here the feature pipeline is a consumer whose inputs are about to change, and the organization is a different fictional one.

Everything ran on macOS 26.7.1 with Python 3.12.14 and NumPy 2.2.3, with no other dependencies. Both platforms are simulated in memory. The embedding is a deterministic stub that hashes character trigrams into 256 dimensions, not a learned model, and the propensity model is a fixed formula standing in for a trained one. The last sections list what follows from those choices.

Requirements and constraints

The team writes four requirements before it designs anything.

• No interruption. Search keeps serving and the weekly campaign keeps running while the migration proceeds.
• Equivalence defined by each consumer. The goal is not byte-identical output. It is differences that each consumer’s owner has agreed are immaterial, stated as numbers before the comparison is run.
• Evidence before each step. No consumer moves because the new jobs finished without errors.
• A way back. Until an observation period ends, each consumer can return to the legacy outputs within one daily cycle.

Three constraints shape the design. Both platforms read the same exports from the source systems, so any difference between their outputs comes from the pipelines and not from different inputs. The legacy batch keeps running through the observation period, which means paying for two platforms during it. And the consumer owners are not the migration team, so what the consumers rely on has to be written down instead of assumed. The source systems also have schedules the team may not know completely, a point that matters later.

Dataset	Grain and content	Read by
dim_product	One row per SKU: surrogate product_id, title, description, category, tags, list price	Search index build; click logs and saved lists store product_id
fct_order_line	One row per (order_id, line_no): quantity, price, discount, line amount in cents, order date	Feature pipeline
feat_customer_asof	One row per customer per as-of day: orders and spend in 30 days, days since last order, average discount in 90 days	Reminder campaign model

The synthetic source holds 600 products, 1,500 customers, and 33,367 order lines over a calendar numbered from day 0 to day 135. The legacy platform assigns product_id from a sequence in the order products first arrived and keeps that mapping as a registry.

What the two consumers depend on

The search index depends on text, and clients depend on identifiers

An embedding is a vector that a model produces from an input, and similarity between vectors is only meaningful among vectors produced the same way. The search service builds one text string per product and embeds it. Callers then receive product_id values, which they store. As Document Pipelines for Retrieval put it, the embedding model is part of the schema; the text that feeds it is as well, since a change to the text changes the vector even when the model stays fixed.

The stub below plays the model’s role. It is deliberately case-sensitive and sensitive to Unicode form: it hashes the raw characters with no normalization. Whether a real model reacts to the same differences depends on the tokenizer in front of it, and some tokenizers normalize text first. That has to be measured on the real model; the stub makes the mechanism visible.

DIM = 256

def product_text(p):
    return f"{p['title']}. {p['description']} Category: {p['category']}. Tags: {p['tags_text'].replace('|', ', ')}"

def embed_text(t):
    v = np.zeros(DIM)
    for i in range(len(t) - 2):
        h = zlib.crc32(t[i:i + 3].encode("utf-8"))
        v[h % DIM] += 1.0 if (h >> 9) & 1 else -1.0
    n = np.linalg.norm(v)
    return v / n if n else v

def search(index, q, k=10):
    ids, M = index
    s = np.round(M @ embed_text(q), 12)
    order = np.lexsort((ids, -s))[:k]
    return [int(ids[i]) for i in order]

The campaign depends on feature values near a threshold

A feature is an input attribute with a definition, and the definition includes time basis and missing-value treatment. The campaign consumer reads four features, adapts them with its own missing-value rules, scores each customer with fixed coefficients, and sends a reminder when the score reaches 0.5. The intercept was chosen so that about 30% of customers are flagged on the first day. Everything the migration can break reaches the campaign through one number, so the comparison tracks it directly: the flip rate, the share of customers whose flag differs between legacy and new features.

def adapt(f):
    return (f["orders_30d"], f["spend_30d_cents"] / 100,
            0 if f["days_since_last_order"] is None else min(f["days_since_last_order"], 120),
            0.0 if f["avg_discount_pct_90d"] is None else f["avg_discount_pct_90d"])

def propensity(f, b=-3.1):
    o, s, d, a = adapt(f)
    z = b + 0.35 * o + 0.0015 * s - 0.02 * d + 0.01 * a
    return 1 / (1 + math.exp(-z))

def campaign_flag(f): return propensity(f) >= 0.5

The adapter matters. If the new pipeline emits a missing value where legacy emitted a sentinel, this consumer silently substitutes zero days since the last order, which scores as a very recent buyer. That is training-serving skew in the sense of the glossary entry: the field name is unchanged and its meaning is not.

Six ways the output changes without changing the schema

A rewrite changes the implementation of every transformation, and each implementation carries defaults that the old one carried differently. The first build of the new platform differs from legacy in six ways. They are typical of what a rewrite introduces; they were chosen for this case and labeled with letters so each can be switched on alone later.

Code	Legacy behavior	New platform, first build	What a consumer sees
K	product_id comes from a sequence in arrival order and is never reassigned	product_id is the row number over SKUs sorted alphabetically	All 600 ids point to a different product; click logs and saved lists now refer to the wrong items
R	Line amount rounded half up in decimal arithmetic	Line amount from round() on a float	2,036 of 33,367 lines (6.1%) are one cent lower
T	Order date is the store-local date (fixed offset UTC-6 in this model)	Order date is the UTC date	7,479 lines (22.4%) have a different date, which moves 30-day window boundaries and days since last order
S	Text is normalized to Unicode NFC, non-breaking spaces become spaces, whitespace is collapsed	Text is only stripped at the ends	The text of 109 products differs by bytes though most render identically
N	Missing category becomes UNCATEGORIZED, missing discount becomes 0, no orders ever becomes 999 days	Missing values stay NULL	38 products have a different category string in the embedded text; features contain NULLs and a different average discount
O	Tags are sorted alphabetically	Tags keep source order	The embedded text of 465 products differs in tag order

def clean_text(s):
    s = unicodedata.normalize("NFC", s).replace(" ", " ")
    return re.sub(r"\s+", " ", s).strip()

def amount_half_up(qty, price, disc):
    return int((Decimal(qty) * price * (Decimal(100) - Decimal(disc or 0)) / 100).quantize(Decimal(1), ROUND_HALF_UP))

def amount_float(qty, price, disc):
    return round(qty * price * (1 - (disc or 0) / 100))

def build_products(src, impl, faults=()):
    reg = src["registry"]; out = []
    ordered = sorted(p["sku"] for p in src["products"])
    ids = {s: i + 1 for i, s in enumerate(ordered)} if (impl == "new" and "K" in faults) else reg
    for p in src["products"]:
        if impl == "legacy" or "S" not in faults:
            title, desc = clean_text(p["title"]), clean_text(p["description"])
        else:
            title, desc = p["title"].strip(), p["description"].strip()
        if impl == "legacy" or "N" not in faults: cat = p["category"] or "UNCATEGORIZED"
        else: cat = p["category"]
        tags = sorted(p["tags"]) if (impl == "legacy" or "O" not in faults) else list(p["tags"])
        out.append(dict(product_id=ids[p["sku"]], sku=p["sku"], title=title, description=desc,
                        category=cat, tags_text="|".join(tags), list_price_cents=p["list_price_cents"]))
    return out

def build_lines(src, impl, faults=()):
    out = []
    for l in src["lines"]:
        new = impl == "new"
        amt = (amount_float if new and "R" in faults else amount_half_up)(l["qty"], l["unit_price_cents"], l["discount_pct"])
        local_date = (l["ordered_at_utc"] + STORE_OFFSET).date()
        odate = l["ordered_at_utc"].date() if (new and "T" in faults) else local_date
        disc = l["discount_pct"] if (new and "N" in faults) else (l["discount_pct"] or 0)
        out.append(dict(order_id=l["order_id"], line_no=l["line_no"], customer_id=l["customer_id"], sku=l["sku"],
                        qty=l["qty"], unit_price_cents=l["unit_price_cents"], discount_pct=disc,
                        line_amount_cents=amt, order_date=odate,
                        ordered_day_utc=l["ordered_day_utc"], ingest_day=l["ingest_day"]))
    return out

The faults argument is scaffolding of the simulation: a real rewrite does not come with labels, and the labels let each difference be switched on alone. The code also shows where each difference sits. Three of them deserve a closer look because they are easy to dismiss.

Rounding. Python’s documentation says round() on a float rounds ties to the even neighbor and that decimal fractions often cannot be represented exactly as binary floats. Legacy rounded ties up. A tie to the cent occurs whenever quantity, price, and discount produce a half cent, and the two rules then agree about half the time:

qty=1 price=1995 disc=10%  exact=1795.5  half-up=1796  float round()=1796
qty=1 price=1005 disc=10%  exact=904.5  half-up=905  float round()=904
qty=3 price=1995 disc=15%  exact=5087.25  half-up=5087  float round()=5087
qty=5 price=1995 disc=5%  exact=9476.25  half-up=9476  float round()=9476
qty=1 price=1985 disc=5%  exact=1885.75  half-up=1886  float round()=1886
lines differing: 2036 of 33367 Counter({0: 31331, 1: 2036})

The first line is a tie on which both rules choose 1796, because 1796 is even. The second is a tie on which they split, 905 against 904. Across the data, 2,036 lines differ, each by exactly one cent.

Unicode form. Two strings can render identically and compare unequal. Unicode defines canonical equivalence between a precomposed character and a base character followed by a combining mark, and the NFC and NFD normalization forms (Unicode Standard Annex 15) map equivalent strings to one representation. A source system that exports the decomposed form changes the characters the embedding stub sees. See Unicode for the encoding background. On the stub:

Zürnholt clamp | Zürnholt clamp | equal: False | lengths: 14 15
code points: ['0x5a', '0xfc', '0x72'] ['0x5a', '0x75', '0x308', '0x72']
cosine of the stub embeddings: 0.8006

The two names look the same, differ in length, and have different code points; the stub embeddings of this short string have a cosine of 0.80. A longer product text dilutes the effect, but the mechanism is the same.

Surrogate keys. A surrogate key identifies a row without business meaning, and its value is part of what consumers rely on once they store it. Regenerating surrogate keys by a different rule is a common side effect of a rewrite. The article on Dimensional Modeling and Historical Restatement covers durable keys; the point here is that key stability must be a stated promise, because nothing in a column’s type reveals it.

A data contract that names what the consumers rely on

What the contract contains

A data contract states what a producer supplies and what consumers may rely on, which covers meaning as well as structure. Data Pipeline CI/CD and Environment Promotion separates three layers of promise: structure, semantics, and behavior. The Halden contract adds key stability and freshness, because the consumers depend on both. The glossary entry on schema evolution and compatibility explains why the structural layer alone fails: a column can keep its type while its unit or time basis changes.

• Structure: column names, types, and nullability.
• Grain: the columns that identify a row, checked for duplicates.
• Semantics: the unit, rounding rule, time basis, and null policy of each column, written as a rule precise enough to recompute from the source.
• Key stability: the identifiers that consumers store, and the promise that they are not reassigned.
• Freshness: the longest acceptable lag between the source and the build.

The contract as code, and the check that enforces it

The contract lives in the repository next to the pipeline and runs in CI and in production. Version 1 is what the team could write down from the schema and the legacy job’s documentation.

CONTRACT_V1 = {
  "contract": "halden.catalog_orders_features", "version": 1,
  "datasets": {
    "dim_product": {
      "grain": ["sku"],
      "columns": {"product_id": ["int", False], "sku": ["str", False], "title": ["str", False],
                  "description": ["str", False], "category": ["str", False],
                  "tags_text": ["str", False], "list_price_cents": ["int", False]},
      "key_stability": "product_id for every sku equals the frozen registry; ids are never reused",
    },
    "fct_order_line": {
      "grain": ["order_id", "line_no"],
      "columns": {"order_id": ["int", False], "line_no": ["int", False], "customer_id": ["int", False],
                  "sku": ["str", False], "qty": ["int", False], "unit_price_cents": ["int", False],
                  "discount_pct": ["num", False], "line_amount_cents": ["int", False],
                  "order_date": ["date", False], "ordered_day_utc": ["int", False],
                  "ingest_day": ["int", False]},
      "semantics": {"line_amount_cents": "round half up of qty*unit_price_cents*(100-discount_pct)/100",
                    "order_date": "calendar date at the store offset UTC-6"},
    },
    "feat_customer_asof": {
      "grain": ["customer_id", "as_of_day"],
      "columns": {"customer_id": ["int", False], "as_of_day": ["int", False], "orders_30d": ["int", False],
                  "spend_30d_cents": ["int", False], "days_since_last_order": ["int", False],
                  "avg_discount_pct_90d": ["num", False]},
      "semantics": {"days_since_last_order": ">= 0; 999 means no order ever",
                    "avg_discount_pct_90d": "mean over lines in 90 days, missing discount counted as 0"},
    },
  },
  "freshness": {"max_build_lag_hours": 6},
}

The checker reads the structural parts of the contract directly. For semantics it recomputes the promised value from the source records with the rule the contract states. That choice matters: a check that reuses the pipeline’s own logic would agree with the pipeline whatever the pipeline did, a point made in Data Quality Invariants and Business Reconciliation about shared errors.

def check(contract, tables, src, build_lag_hours):
    """Return {check_name: violation_count} for one platform's output."""
    out = {}
    for name, spec in contract["datasets"].items():
        rows = tables[name]
        out[f"{name}.schema"] = schema_violations(name, rows, spec)
        out[f"{name}.grain"] = grain_violations(rows, spec["grain"])
    reg = src["registry"]
    out["dim_product.key_stability"] = sum(reg[p["sku"]] != p["product_id"] for p in tables["dim_product"])
    srcline = {(l["order_id"], l["line_no"]): l for l in src["lines"]}
    amt = day = 0
    for r in tables["fct_order_line"]:
        s = srcline[(r["order_id"], r["line_no"])]
        amt += r["line_amount_cents"] != sim.amount_half_up(s["qty"], s["unit_price_cents"], s["discount_pct"])
        day += r["order_date"] != (s["ordered_at_utc"] + sim.STORE_OFFSET).date()
    out["fct_order_line.line_amount_semantics"] = amt
    out["fct_order_line.order_date_semantics"] = day
    if contract["version"] >= 2:
        def bad_text(s): return s != sim.clean_text(s)
        out["dim_product.text_normalization"] = sum(bad_text(p["title"]) or bad_text(p["description"]) for p in tables["dim_product"])
        out["dim_product.tags_sorted"] = sum(p["tags_text"].split("|") != sorted(p["tags_text"].split("|")) for p in tables["dim_product"])
    f = tables["feat_customer_asof"]
    out["feat_customer_asof.sentinel_semantics"] = sum(r["days_since_last_order"] is not None and r["days_since_last_order"] < 0 for r in f)
    out["freshness.build_lag"] = int(build_lag_hours > contract["freshness"]["max_build_lag_hours"])
    return out

Run it on the incumbent first

A contract describes what consumers may rely on today, so the legacy outputs must satisfy it. If they do not, the contract is a wish and the first failure will be the contract’s. Then the same checks run on the first build of the new platform, with all six differences present. Output of the run, for the output tables as of day 112:

legacy             contract v1: {}
legacy             contract v2: {}
new (all faults)   contract v1: {'dim_product.schema': 38, 'fct_order_line.schema': 16636, 'feat_customer_asof.schema': 263, 'dim_product.key_stability': 600, 'fct_order_line.line_amount_semantics': 2036, 'fct_order_line.order_date_semantics': 7479}
new (all faults)   contract v2: {'dim_product.schema': 38, 'fct_order_line.schema': 16636, 'feat_customer_asof.schema': 263, 'dim_product.key_stability': 600, 'fct_order_line.line_amount_semantics': 2036, 'fct_order_line.order_date_semantics': 7479, 'dim_product.text_normalization': 109, 'dim_product.tags_sorted': 465}

Legacy passes both versions. The new platform fails version 1 on five of the six differences, and the counts tell what broke: all 600 product ids moved, 2,036 line amounts and 7,479 order dates disagree with the stated rules, and NULLs appear in columns that are declared non-null (38 categories, 16,636 discounts, 263 feature values). Two differences are silent under version 1. Nobody wrote down that descriptions are normalized or that tags are sorted, because nobody knew the embedding text depended on either. Version 2 adds those two clauses, and it fails on exactly the 109 and 465 rows. Where the clauses come from is the next section.

Dual running and reconciliation

Why run both platforms

The contract holds what the team knows to ask. Dual running finds differences it has not thought to ask about, by running the legacy and new pipelines on the same inputs and comparing outputs. The alternatives each give up something:

Approach	Evidence it provides	What it gives up
Cut over in one step	None beyond the new job succeeding	Any silent difference is found by consumers, and rollback means restoring the old platform under pressure
Replay history through the new pipeline and compare	Behavior on recorded data	Live behavior: the source’s current schedules, late arrivals, and quirks that history has already smoothed over
Dual running on live daily inputs	Behavior on current inputs for as long as the window lasts, and a ready fallback	Two platforms’ compute and storage, comparison tooling, and a later finish
Move one consumer at a time	Consumer-specific evidence and a small blast radius	A longer migration and a period in which consumers read different platforms

Halden combines the last two. Dual running produces the evidence that a consumer may move, and the consumers move separately so that each has its own gate and its own way back.

Two disciplines keep the comparison meaningful. Both pipelines read the same frozen snapshot of the source exports for each day, which applies the earlier point from Building Incremental and Idempotent Pipelines that a logical operation is bound to retained input. And comparison is by natural key (SKU, order and line number, customer and as-of day), never by the surrogate id, because the surrogate id is one of the things under test.

The reconciliation ladder

Data Quality Invariants and Business Reconciliation orders reconciliation by depth: row counts, totals, key sets, and row-level values, each catching faults the shallower level cannot. The comparison below applies it to the day-112 outputs of the legacy and the first-build platform.

def recon(leg, new):
    """Reconciliation ladder between two platforms' outputs. leg/new: dict of tables."""
    res = {}
    for t in ("dim_product", "fct_order_line", "feat_customer_asof"):
        res[f"L1 rows {t}"] = (len(leg[t]), len(new[t]))
    res["L2 sum line_amount_cents"] = (sum(r["line_amount_cents"] for r in leg["fct_order_line"]),
                                       sum(r["line_amount_cents"] for r in new["fct_order_line"]))
    res["L2 sum spend_30d_cents"] = (sum(r["spend_30d_cents"] for r in leg["feat_customer_asof"]),
                                     sum(r["spend_30d_cents"] for r in new["feat_customer_asof"]))
    ids = lambda t: {r["product_id"] for r in t["dim_product"]}
    res["L3 product_id key set differences"] = len(ids(leg) ^ ids(new))
    pairs = lambda t: {(r["product_id"], r["sku"]) for r in t["dim_product"]}
    res["L3 (product_id, sku) pairs differing"] = len(pairs(leg) ^ pairs(new))
    lk = lambda t: {(r["order_id"], r["line_no"]) for r in t["fct_order_line"]}
    res["L3 order-line key set differences"] = len(lk(leg) ^ lk(new))
    # L4: compare by natural key, column by column
    def cols_diff(name, key, cols):
        a = {tuple(r[k] for k in key): r for r in leg[name]}
        b = {tuple(r[k] for k in key): r for r in new[name]}
        out = {}
        for c in cols:
            out = sum(a[k] != b[k] for k in a.keys() & b.keys())
        out["(any column)"] = sum(rowhash(a[k], cols) != rowhash(b[k], cols) for k in a.keys() & b.keys())
        return out
    res["L4 dim_product"] = cols_diff("dim_product", ("sku",), ["product_id", "title", "description", "category", "tags_text", "list_price_cents"])
    res["L4 fct_order_line"] = cols_diff("fct_order_line", ("order_id", "line_no"), ["line_amount_cents", "order_date", "discount_pct"])
    return res

L1 rows dim_product (600, 600)
L1 rows fct_order_line (33367, 33367)
L1 rows feat_customer_asof (1500, 1500)
L2 sum line_amount_cents (805060751, 805058715)
L2 sum spend_30d_cents (166086031, 167082576)
L3 product_id key set differences 0
L3 (product_id, sku) pairs differing 1200
L3 order-line key set differences 0
L4 dim_product {'product_id': 600, 'title': 0, 'description': 109, 'category': 38, 'tags_text': 465, 'list_price_cents': 0, '(any column)': 600}
L4 fct_order_line {'line_amount_cents': 2036, 'order_date': 7479, 'discount_pct': 16636, '(any column)': 21919}

Reading the output from the top shows each level’s blind spot.

• Row counts agree on every table. Counts are the cheapest check and they pass.
• Line-amount totals differ by 2,036 cents out of about 805 million, a relative difference of 0.00025%. Customer spend totals differ by 0.60%, which comes from dates moving across the 30-day boundary. A tolerance of 0.1%, a common default, would pass the first and fail the second.
• The product id key sets are identical, because the new ids are a permutation of 1 to 600. A key-set check passes while every id refers to a different product. Only the pair check, which compares each (product_id, SKU) pair, reports 1,200 differing pairs (600 on each side).
• The column-level comparison finds what the others missed, and it attributes each difference to a column: 109 descriptions, 38 categories, 465 tag strings, 2,036 amounts, 7,479 dates, 16,636 discounts. At scale, comparing every row is expensive, and a chunked checksum reduces the transfer by comparing a row count and a content hash for each key range first, then looking at rows only in ranges that differ.

Counts, totals, and key sets all passed a plausible threshold and the data was wrong in six ways. This is the argument for comparing at the key and column level and for reading the result by column.

Comparing what the consumers actually use

A column difference matters to a consumer only through what the consumer computes from it. The comparison therefore continues one step past the tables, to the vectors and the decisions.

Embeddings: vector similarity and retrieval overlap

Two measures cover an index. For each product, the cosine between its legacy and new vectors (matched by SKU) says how far the text change moved it; see the embedding entry for what cosine compares. For a set of 80 queries, the overlap at 10 says how much of the top-10 result list is shared between the two indexes. The overlap is computed twice: by product_id, which is what clients see, and by SKU, which isolates content.

Measure (first-build platform, all six differences)	Value
Products whose vector is identical	18.7%
Lowest cosine to the legacy vector	0.874
First percentile of cosine	0.901
Top-10 overlap by product_id, mean (minimum)	0.024 (0.0)
Top-10 overlap by SKU, mean (minimum)	0.946 (0.8)

The id-based overlap is close to zero because of the key change alone; every returned id points to a different product. The SKU-based overlap removes that effect and shows the content effect: on average 5.4% of top-10 slots differ, and the worst query shares eight of ten results. Neither number says whether the new ranking is worse. It says it is different, and the difference is measured against a baseline. For a real embedding model that baseline needs a noise floor: embed the same text repeatedly, on the hardware and software the production path uses, and take the observed spread as the smallest difference that means anything. The stub is deterministic, so its floor is zero.

Features: per-feature mismatch and decision flips

For features, the comparison reports the mismatch rate for each feature separately, since one noisy feature can hide a defect in another, and the flip rate of the campaign decision. This follows the parity-failure reading in Feature Pipelines and Point-in-Time Correctness.

{'orders_30d': 0.015333333333333332, 'spend_30d_cents': 0.214, 'days_since_last_order': 0.266, 'avg_discount_pct_90d': 0.9606666666666667, 'flip_rate': 0.010666666666666666, 'max_abs_spend_diff_cents': 148349}
legacy flag rate 0.31133333333333335

The mismatch rates are large (21% of customers have a different 30-day spend, 27% a different days-since-last-order, 96% a different average discount) while the flip rate is 1.07%, 16 of 1,500 customers. The model’s threshold absorbs most of the movement. The 1.07% is not harmless for that reason: those 16 customers receive a different decision, and the rate is above the 0.5% the campaign owner will allow.

Tolerances: who sets them and how

A comparison needs a number below which a difference is accepted, and the number cannot be set by whoever wants the migration to pass. Two sources keep it honest. The consumer owner sets it from what the consumer can tolerate, and the migration team measures how the consumer’s output responds to controlled perturbations so the owner can choose with evidence. For the campaign, the experiment perturbs legacy features by a known amount and measures the flip rate (seven as-of days, 10,500 customer-days, one random draw per row, so small differences between neighboring rows are sampling noise).

share of rows within |z|<0.05 / 0.2 / 0.5 of the boundary: [0.0105, 0.0401, 0.1043]
relative noise on spend_30d (uniform +/- eps):
  eps=0.001  flip rate 0.010%
  eps=0.005  flip rate 0.048%
  eps=0.010  flip rate 0.124%
  eps=0.020  flip rate 0.143%
  eps=0.050  flip rate 0.610%
one fewer order in the 30-day window for p of customers:
  p=0.01  flip rate 0.019%
  p=0.03  flip rate 0.076%
  p=0.08  flip rate 0.286%

About 10% of customer-days lie within 0.5 score units of the decision boundary. A random error of 5% in 30-day spend flips 0.61% of decisions; losing one order from the 30-day count for 8% of customers flips 0.29%. The campaign owner sets the budget at 0.5% of customers and records the reason: a reminder sent in error costs little, and a missed reminder costs one lost opportunity per customer. For search, the owner sets mean top-10 overlap by SKU at 0.98 or higher and no query below 0.8, because a changed ranking moves traffic between products.

These tolerances apply to consumers. The contract applies stricter rules to the pipeline, because exactness there is cheap and finance sums the amounts. The rounding difference shows the gap: it produced zero flips and a line-amount total within 0.00025%, yet it breaks the contract on 2,036 lines. A team can handle it by fixing the rounding or by recording an accepted break with an owner, a reason, and an expiry date, as Data Quality Invariants and Business Reconciliation describes. The decision rule below supports both. What it must not do is accept the difference because the consumer metrics happen to pass today.

A tolerance also has to be smaller than the fault it is meant to catch. A 0.1% tolerance on totals cannot see the rounding difference, and a 5% flip tolerance would have waved through every scenario in this case.

What each check catches: injecting the faults one at a time

The combined run shows that something is wrong. It does not show which check would have caught which fault, and a check that never fires is worth little. The matrix below switches on each difference alone, builds the new platform, and records which detectors fire. The detectors are the two contract versions, the four reconciliation levels (totals at a 0.1% tolerance, key sets, id and SKU pairs, and column values), the consumer gates from the previous section, and a business invariant that compares each line’s unit price with the product’s list price. The control row has no fault injected.

def detectors(src, leg, new, queries, lag=2.5):
    d = {}
    d["C1"] = sum(contract.check(contract.CONTRACT_V1, new, src, lag).values()) > 0
    d["C2"] = sum(contract.check(contract.CONTRACT_V2, new, src, lag).values()) > 0
    r = compare.recon(leg, new)
    d["L1"] = any(r[k][0] != r[k][1] for k in r if k.startswith("L1"))
    d["L2"] = any(abs(r[k][0] - r[k][1]) / r[k][0] > 0.001 for k in r if k.startswith("L2"))
    d["L3k"] = r["L3 product_id key set differences"] > 0 or r["L3 order-line key set differences"] > 0
    d["L3p"] = r["L3 (product_id, sku) pairs differing"] > 0
    f = compare.feature_compare(leg["feat_customer_asof"], new["feat_customer_asof"])
    d["L4"] = (r["L4 dim_product"]["(any column)"] > 0 or r["L4 fct_order_line"]["(any column)"] > 0
               or any(f > 0 for c in ("orders_30d", "spend_30d_cents", "days_since_last_order", "avg_discount_pct_90d")))
    e = compare.embedding_compare(leg["dim_product"], new["dim_product"], queries)
    d["EMB"] = e["overlap_id_mean"] < 0.98 or e["overlap_sku_mean"] < 0.98 or e["overlap_sku_min"] < 0.8
    d["FEAT"] = f["flip_rate"] > 0.005
    d["INV"] = price_invariant(new) > 0
    return d, r, e, f

Injected	Contract v1	Contract v2	Counts	Totals (0.1%)	Key sets	Id-SKU pairs	Column values	Embedding gate	Flip gate	Price invariant
K surrogate keys	yes	yes	–	–	–	yes	yes	yes	–	–
R float rounding	yes	yes	–	–	–	–	yes	–	–	–
T UTC order date	yes	yes	–	yes	–	–	yes	–	–	–
S text normalization	–	yes	–	–	–	–	yes	–	–	–
N NULL handling	yes	yes	–	–	–	–	yes	–	yes	–
O tag order	–	yes	–	–	–	–	yes	yes	–	–
All six	yes	yes	–	yes	–	yes	yes	yes	yes	–
None (control)	–	–	–	–	–	–	–	–	–	–

The per-fault consumer measurements behind the gate columns were: flip rate 0.00% for K, R, S, and O, 0.20% for T, and 0.73% for N; mean top-10 overlap by SKU 0.997 for K, 1.000 for R and T, 0.984 for S, 0.990 for N, and 0.959 for O.

Reading the matrix gives the following.

• No single check covers the set. Column-level comparison fires on all six, but only after someone decides to compare that column and reads the result. The contract fires on four of six under version 1 and on all six under version 2.
• Counts and key sets fire on nothing. They are worth running because they catch gross loss and duplication, which this scenario did not contain.
• The id-SKU pair check is what finds K. The key-set check cannot, since the new ids are a permutation of the old.
• The consumer gates are narrow by design. The flip gate fires only for N. T moves 0.20% of decisions, which is under the 0.5% budget, and a team relying on the gate alone would have accepted a platform whose dates are wrong on 22% of lines. The embedding gate fires for K and O but not for S or N: 109 and 38 changed texts barely move an 80-query overlap average. Those are real changes to what the index contains, and only the data-level checks see them.
• Text differences need the data-level check. S and O, the two that version 1 missed, were found by comparing what feeds the embeddings. Both became contract clauses.

The matrix has the limits of its method. Faults were injected one at a time, so interactions are not measured. It uses one as-of day, one generated dataset, one query set, and thresholds chosen for the case. A tie between a fault and a detector in this table says the detector can see that fault at that size, not that it will see it in another dataset.

Turning findings into clauses and fixes

Each difference the comparison finds has two destinations: a fix in the new platform, and a clause in the contract so that the difference cannot return unnoticed. Version 2 of the contract adds the two clauses that version 1 lacked.

CONTRACT_V2["datasets"]["dim_product"]["semantics"] = {
    "title,description": "Unicode NFC, no U+00A0, single spaces, no leading or trailing space",
    "tags_text": "tags sorted alphabetically, joined by |"}

The fixes follow the expand-and-contract discipline from Data Pipeline CI/CD and Environment Promotion where consumers are involved. For keys, the new platform loads the frozen registry and keeps every existing id, and assigns ids for new SKUs above the registry’s maximum, so the promise that an id is never reused survives the rewrite. For rounding and dates, the new pipeline adopts the decimal half-up rule and the store-local date conversion that the contract states. For text, it shares the legacy normalization function instead of re-implementing it. For NULLs, it applies the documented defaults. The repaired platform is the control row of the matrix: every detector is quiet.

The cutover plan and its decision rule

Stages

A cutover moves live work from the old path to the new one, and for Halden it happens in stages with an exit rule for each. The stages apply to each consumer. In the run below the two consumers entered canary on the same day because their evidence was ready, but they have separate routing entries and gates and can be moved back independently.

Stage	Who reads the new platform	Exit rule
0. Dual run	Nobody; legacy serves both consumers	At least 7 days; every known source event observed; no contract violations, unexplained column differences, or id-SKU pair differences; search overlap and flip rate within tolerance
1. Canary	10% of customers for the campaign, 10% of searches	Two clean days on that population
2. Full	All traffic of that consumer; legacy keeps running as a shadow	Entered on a clean canary
3. Observation	As stage 2, for 14 days	Daily flip rate below the hold limit; rollback above 1.0%; search overlap in tolerance
4. Decommission	Legacy stops	Observation complete, a recorded decision to give up the fallback

The rule as code

The rule takes an evidence record and returns one of three verdicts. Promote means every gate holds and the next stage may begin. Hold means the stage stays as it is while the listed issues are fixed or more evidence accumulates. Rollback exists only after a consumer is live on the new platform. The thresholds are the ones the owners set above. Each verdict carries its reasons, so a hold is an actionable list. The structure follows the three outcomes of a release gate in Prompt and Model Promotion with Evaluation Gates.

RULES = dict(min_window_days=7, known_events={"nightly_load", "weekly_refresh"},
             search_overlap_mean=0.98, search_overlap_min=0.80,
             flip_hold=0.005, flip_rollback=0.010)

@dataclass
class Evidence:
    window_days: int
    events_observed: set
    contract_blocking: int
    pair_mismatch: int
    unexplained_diffs: int
    search_overlap_mean: float
    search_overlap_min: float
    flip_rate: float
    notes: list = field(default_factory=list)

def decide(ev, rules, post_cutover=False):
    hold, rollback = [], []
    if ev.window_days < rules["min_window_days"]:
        hold.append(f"window {ev.window_days}d < {rules['min_window_days']}d")
    missing = rules["known_events"] - ev.events_observed
    if missing: hold.append(f"events not observed in window: {sorted(missing)}")
    if ev.contract_blocking: hold.append(f"{ev.contract_blocking} contract violations")
    if ev.pair_mismatch: hold.append(f"{ev.pair_mismatch} (product_id, sku) pairs differ")
    if ev.unexplained_diffs: hold.append(f"{ev.unexplained_diffs} unexplained column differences")
    if ev.search_overlap_mean < rules["search_overlap_mean"] or ev.search_overlap_min < rules["search_overlap_min"]:
        hold.append(f"search overlap mean {ev.search_overlap_mean:.3f} / min {ev.search_overlap_min:.2f}")
    if ev.flip_rate > rules["flip_hold"]: hold.append(f"decision flip rate {100*ev.flip_rate:.2f}% > {100*rules['flip_hold']:.1f}%")
    if post_cutover and ev.flip_rate > rules["flip_rollback"]:
        rollback.append(f"decision flip rate {100*ev.flip_rate:.2f}% > {100*rules['flip_rollback']:.1f}% after cutover")
    if rollback: return "ROLLBACK", rollback + hold
    if hold: return "HOLD", hold
    return "PROMOTE", ["all gates satisfied"]

def observe(daily_flip, search_overlap_mean, rules, observation_days=14):
    """Post-cutover rule: evaluated daily on the population served by the new platform."""
    for i, f in enumerate(daily_flip, 1):
        if f > rules["flip_rollback"]:
            return "ROLLBACK", f"day {i} of observation: flip rate {100*f:.2f}% > {100*rules['flip_rollback']:.1f}%"
        if f > rules["flip_hold"]:
            return "HOLD", f"day {i} of observation: flip rate {100*f:.2f}% > {100*rules['flip_hold']:.1f}%, no further promotion"
    if search_overlap_mean < rules["search_overlap_mean"]:
        return "ROLLBACK", "search overlap below limit"
    if len(daily_flip) < observation_days:
        return "CONTINUE", f"{len(daily_flip)} of {observation_days} observation days clean"
    return "PROMOTE", f"{observation_days} observation days clean: legacy may be decommissioned"

Two parts of the rule come from earlier sections. known_events is the inventory of source events that a dual-run window must contain, and the rule holds if the window has not observed one of them. waivers, in the code that builds the evidence, remove a named contract check and column difference from the blocking counts, which is how an accepted break is represented; the expiry date belongs to the process around the set.

Running the scenarios

Each scenario builds the evidence for a seven-day window (days 106 to 112) from the simulated platforms and applies the rule.

S1 dual-run exit, first build (K R T S N O)
  window 7d  contract 27626  pair 1200  col-diffs 26151  overlap 0.946/0.80  flip 1.01%
  -> HOLD: 27626 contract violations; 1200 (product_id, sku) pairs differ; 26151 unexplained column differences; search overlap mean 0.946 / min 0.80; decision flip rate 1.01% > 0.5%
S2 after fixing K T S N O, R remains
  window 7d  contract 2036  pair 0  col-diffs 2036  overlap 1.000/1.00  flip 0.00%
  -> HOLD: 2036 contract violations; 2036 unexplained column differences
S2b R waived (owner, expiry recorded)
  window 7d  contract 0  pair 0  col-diffs 0  overlap 1.000/1.00  flip 0.00%
  -> PROMOTE: all gates satisfied
S3 all differences fixed, dual-run exit
  window 7d  contract 0  pair 0  col-diffs 0  overlap 1.000/1.00  flip 0.00%
  -> PROMOTE: all gates satisfied
S4 canary: 10% customers, 14 of 80 queries
  window 7d  contract 0  pair 0  col-diffs 0  overlap 1.000/1.00  flip 0.00%
  -> PROMOTE: all gates satisfied

Scenario S1 is the first build. The rule holds and lists five reasons, so the team knows what to fix without reading raw diffs. S2 is the repaired platform with only the rounding difference left. The consumer metrics are perfect (overlap 1.000, flip rate 0.00%), yet the verdict is still hold, because the contract and the column comparison both flag 2,036 lines. The contract is stricter than the consumers, on purpose. S2b records the rounding difference as a waived break and the rule promotes. S3 fixes it and promotes. S4 applies the rule to the canary population of 10% of customers and 14 of the 80 queries, and also promotes.

One property of S4 is easy to miss. A canary population of 150 customers resolves a flip rate in steps of 0.67% per customer, so a 0.5% threshold means that a single flipped customer causes a hold. That is a reasonable stance for a first step, but the canary threshold should be stated as a count of customers, not copied from the full-population percentage.

Routing and rollback

Rollback is a pointer change. Each consumer reads through a router that says which platform serves it and for which share of keys, so moving a consumer forward or back is one recorded operation. The share is decided by a hash of the key, so a customer stays on the same platform for the whole canary.

class Router:
    """Which platform serves each consumer; flipping this pointer is the rollback."""
    def __init__(self):
        self.state = {"search": ("legacy", 0.0), "features": ("legacy", 0.0)}; self.log = []
    def set(self, consumer, target, share, day, why):
        self.state[consumer] = (target, share)
        self.log.append(f"day {day}: {consumer:8s} -> {target:6s} share {share:4.0%}  ({why})")
    def source_for(self, consumer, key):
        target, share = self.state[consumer]
        return target if target == "new" and zlib.crc32(f"{consumer}:{key}".encode()) % 100 < share * 100 else "legacy"

The pointer flip is the easy part of a migration fallback. It works only while the legacy outputs are still being produced and still trustworthy, so legacy runs through the observation period, and decommissioning is a recorded decision that gives up the way back. The glossary entry on rollback adds the second condition: the version you return to must still work with current data and interfaces. Things the consumers did on the new platform, such as sending a campaign message, are not undone by the flip.

The failure the window missed

The dual-run window passed, and the cutover began. Both consumers entered canary on day 113 and moved to full on day 115. The rule watches the flip rate between what the new platform serves and what the legacy shadow computes for the same customers.

What the team did not know is that 8% of customers belong to partner stores, and those stores send their orders in batches every 16 days instead of daily. The batches arrived on days 100 and 116. The legacy job rebuilds from everything that has arrived, so it counts the late rows. The new incremental pipeline loads rows with a one-day lookback and does not revisit older ones, a bounded correction policy of exactly the kind Building Incremental and Idempotent Pipelines warns about: a row that arrives late carrying an old timestamp falls outside it. The dual-run window ran from day 106 to 112, between two batches, and saw nothing.

def visible(l, asof, impl, lookback_fault):
    if l["ingest_day"] > asof: return False
    if impl == "new" and lookback_fault and l["ingest_day"] > BACKFILL_END:
        return l["ingest_day"] - l["ordered_day_utc"] <= 1   # 1-day lookback: older arrivals never revisited
    return True

The matrix would have said the same: with the lookback defect on, every detector is quiet at as-of day 112, and at day 117 the totals, the column values, and the flip gate all fire (flip rate 1.27% in that run). The post-cutover observation caught it the day the batch landed:

day 113  flip  0.00%  CONTINUE  1 of 14 observation days clean
day 114  flip  0.00%  CONTINUE  2 of 14 observation days clean
day 115  flip  0.00%  CONTINUE  3 of 14 observation days clean
day 116  flip  1.67%  ROLLBACK  day 4 of observation: flip rate 1.67% > 1.0%
customers whose campaign flag differed from legacy on the rollback day: 25
day 113: search   -> new    share  10%  (canary after dual-run exit PROMOTE)
day 113: features -> new    share  10%  (canary after dual-run exit PROMOTE)
day 115: search   -> new    share 100%  (two clean canary days)
day 115: features -> new    share 100%  (two clean canary days)
day 116: features -> legacy share   0%  (day 4 of observation: flip rate 1.67% > 1.0%)
{'search': ('new', 1.0), 'features': ('legacy', 0.0)}

On day 116 the flip rate is 1.67%, above the 1.0% rollback limit, and the rule rolls the feature consumer back to the legacy outputs. Search stays on the new platform, whose evidence was unaffected. The rollback bounds the damage to one day: 25 customers had a campaign decision that differed from legacy on day 116 (1.67% of 1,500). If the campaign had already sent messages that morning, rolling back does not recall them, so that part of the cost stays.

The recovery has three parts. The pipeline’s late-arrival handling is fixed so that rows older than the lookback are reprocessed on arrival. The source event inventory gains partner_sync, found now because the data shows batches whose arrival day is more than a day after the order day. And the dual-run gate is re-run against the updated inventory:

S6a original window 106-112, updated event inventory
  window 7d  contract 0  pair 0  col-diffs 0  overlap 1.000/1.00  flip 0.00%
  -> HOLD: events not observed in window: ['partner_sync']
S6b window 113-126 (lookback fixed, contains day-116 sync)
  window 14d  contract 0  pair 0  col-diffs 0  overlap 1.000/1.00  flip 0.00%
  -> PROMOTE: all gates satisfied

With the corrected inventory the original window would have been held (S6a), because it never observed a partner batch. A window from day 113 to day 126, which contains the day-116 batch, passes with the fixed pipeline (S6b). The sequence also shows why the canary did not help here: it lasted two days against a source cycle of sixteen. A canary or window is only as informative as the cycles it spans, which is the point the Google SRE Workbook makes about canaries needing to be large enough and long enough to expose a defect. The observation period exists because the inventory of source events is never complete, a coverage limit that Metadata and Lineage for Change Impact Analysis describes for lineage.

A fault no comparison can see

One more case belongs in the matrix because no comparison between the two platforms can detect it. Suppose one store has sent unit prices in whole dollars instead of cents since day 60. Legacy and new read the same exports, so they agree, row for row, on the wrong values. In the matrix run this is the final row: every reconciliation level, both contracts, and both consumer gates are silent, and only the invariant that compares each line price with the list price fires, on 941 lines in the legacy output and the same 941 in the new one.

def price_invariant(t):
    lp = {p["sku"]: p["list_price_cents"] for p in t["dim_product"]}
    return sum(not (0.5 <= l["unit_price_cents"] / lp[l["sku"]] <= 1.5) for l in t["fct_order_line"])

Dual running tests that the new platform reproduces the old one, and a shared defect passes that test. The business invariants and independent references from Data Quality Invariants and Business Reconciliation answer a different question, and the migration keeps both kinds of check.

Running two platforms through the migration

Some obligations of the dual-run period did not appear in the simulation and have to be planned.

• Freshness. Each consumer’s freshness is measured where the consumer reads, as Data SLIs, SLOs, and Incident Response recommends. The new platform adds a second source of lag during the canary, and the contract’s build-lag clause should be checked on the platform that is serving.
• Deletion. An erasure request received during the migration has to reach both platforms and every derived copy, including embedding vectors. Otherwise a rollback to the platform that missed the deletion restores data that should be gone (Access, Retention, and Erasure Across Data Copies).
• Permissions. If chunk-level permissions are attached to the index, the new platform must carry the same access metadata, and the comparison should include a leakage check as in Preserving Access Controls in Retrieval Data. The simulated index has no permissions.
• Recovery. The new platform needs its own tested restore (Disaster Recovery and Cost Engineering for Data Platforms). The legacy platform is the fallback only while it is maintained, so the date it stops is also the date the fallback ends.
• One change at a time. If the embedding model changes in the same release as the platform, differences in the search results can no longer be attributed. Versioning Models, Data, Prompts, and Evaluation Artifacts treats the model and its inputs as separately versioned artifacts, and the migration should keep them separate: move the platform with the model fixed, then evaluate the model change on the new platform.

What the simulation does not show

• The embedding is a hash of character trigrams. It reacts to whitespace, Unicode form, and tag order more strongly than many real models do, or differently, and says nothing about retrieval quality. The effect sizes of S, N, and O on a real model must be measured.
• The propensity model is a fixed formula with coefficients chosen for this case. A trained model has a different margin distribution, so the flip-rate numbers do not transfer.
• The data are synthetic with a single random seed, and the platforms are in-memory functions. Nothing here measures run time, cost, concurrency, partial failures, or a real orchestrator.
• The faults were injected one at a time, and the six differences were chosen as typical examples. A real rewrite produces differences nobody listed, which is why the comparison layers exist.
• The source-event inventory and the weekly-refresh coverage are modeled simply: the rule counts a window of seven days as covering the nightly and weekly events, and detects partner batches from the data.
• The thresholds are the case’s own. The passing scenarios show the rule behaving as written, not that those thresholds suit another system.

What the integrated cases settle

The integrated cases put topics the earlier articles treat separately under one set of constraints. In this one, contracts, quality checks, lineage, feature and retrieval pipelines, release gates, and recovery all acted on the same three datasets, and the migration was safe only to the extent that each layer covered what the others could not see.

References

• Python documentation: unicodedata, Unicode Database
• Python documentation: built-in functions, round()
• Unicode Standard Annex #15: Unicode Normalization Forms
• Google SRE Workbook: Canarying Releases