Survivorship Rule
A survivorship rule decides which value appears in a merged record when the records being merged disagree. Once entity resolution has judged that three rows describe the same customer, something still has to choose between three phone numbers, and that choice is what survivorship rules make.
Matching tools generally stop short of this. AWS describes its resolution output as “a consolidated data output table with similar records that have a common match ID” — a grouping and an identifier, not an opinion about which address is right. What the combined record says is a separate step, and the organization owns it.
Four bases, each with a failure mode
Most rules in practice rest on one of four claims about why a value should win.
| Basis | The claim | Where it goes wrong |
|---|---|---|
| Source authority | One system is where this field is legitimately maintained — billing for the invoicing address, HR for employment status | Fails silently when the authoritative system is stale because nobody updates it, or when authority was asserted by an architecture diagram rather than by how people actually work |
| Recency | The most recently updated value is the most current | Confuses a real change with a careless edit, and depends on update timestamps that mean the same thing across systems — a bulk migration can make every row in one source look newest |
| Completeness | A populated value beats a null; a fuller address beats a partial one | Longer is not more correct, and a field can be non-null and wrong. Useful as a tiebreaker, weak as a primary rule |
| Verification status | A value confirmed by a process — an address validated against a postal file, an email that received a click — beats an unconfirmed one | Strongest where available and rarely available for every field; verification also expires, so a confirmation needs a date attached to it |
None is right in general, which is the practical point: fields differ in who maintains them and how they change, so a single rule applied to a whole record is often wrong for some of its fields. Decide and check the rule field by field. Using the same rule for every field is fine when one source is the approved authority for all of them. Legal name may come from the system of registration, contact preference from the most recent customer interaction, and credit terms from finance — in the same merged row.
Writing them as ordered fallbacks handles the common case where the preferred source is empty: take the verified value; failing that, the authoritative source’s value; failing that, the most recent non-null. Stating the chain explicitly is what stops the implicit last step — whatever the load order happened to leave — from becoming the rule. The chain assumes an empty value means unknown. Where a source can also clear a field on purpose — a customer withdrawing a phone number — the clear has to arrive as its own state, distinct from missing, and an authoritative clear must not be back-filled from an older, lesser source; otherwise the fallback quietly resurrects what someone deleted.
Why these are business decisions
The question “which system is authoritative for a customer’s billing address” has consequences for who receives invoices and who is responsible when one goes astray. The data alone does not establish that authority; choose it using the actual maintenance workflow, evidence, and accountability for the field. That makes it the kind of decision that needs a named owner per field, a written record of what was decided, and the reasoning kept beside it.
The failure to avoid is a rule that exists only inside a mapping tool. It still governs what everyone downstream sees, and nobody can find it to argue with it — so when a value looks wrong, the discussion becomes an investigation rather than a decision review. The related quality vocabulary is worth borrowing here: what a survivorship rule is really trying to produce is a record that is accurate, complete, and consistent at once, and those goals can conflict, which is precisely why someone has to choose.
Changing a rule changes history
Because the merged record is derived, editing a rule changes what a rerun produces — and it rewrites outputs people have already used where past ranges are recomputed under the new rule, or where a live consolidated view was what past reports read. A report run last quarter and re-run today can differ for no reason visible in the data. Three habits keep this manageable.
- Never overwrite the sources. The merged value is a derived field; the contributing values must remain, or a rule change becomes unrecoverable and a dispute becomes unanswerable.
- Record which source each field came from. A provenance column per mastered field turns “why does it say this” into a lookup, and it is what makes a bad rule visible in aggregate — if one source wins every field, compare that pattern with the approved authority for each field. It may be legitimate when that source is authoritative for all of them, and otherwise it points to a flaw in the rule.
- Version the rules, date the change, and tie each run to its input. Record the rule version, a run identifier, and the input snapshot for every run. A difference between two runs can then be attributed only by comparing like with like: the same snapshot under two rule versions isolates the rule change, and the same rule version over two snapshots isolates the data change. If both changed between the runs, the version and date alone cannot say how much each contributed; rerun one side under the other’s conditions, and use a reconciliation against the sources to check the result.
Manual overrides deserve a mention of their own. When someone corrects a merged value by hand, that correction is a decision that outranks the automated rule for the case it covers — and it should be recorded as one: who approved it, which record and field it applies to, from when, and until what, whether an expiry, a review date, or a later change from the authoritative source that supersedes it. The next run has to apply the decisions currently in force; it must not keep a manual address pinned over the customer’s own later change, and a mistaken override needs a way to be withdrawn. A pipeline that recomputes the golden record from scratch and silently discards human corrections will be found out by the person who made them.
How survivorship fits with matching, implementation styles, and designing for wrong decisions is worked through in One Customer, Five Systems; the record these rules assemble is a golden record.
References
- AWS Entity Resolution User Guide, What is AWS Entity Resolution?
- GOV.UK, Meet the data quality dimensions
Discover more from Insightful Data Lab
Subscribe to get the latest posts sent to your email.