Survivorship Rule
A survivorship rule decides which value appears in a merged record when the records being merged disagree. Once entity resolution has judged that three rows describe the same customer, something still has to choose between three phone numbers, and that choice is what survivorship rules make.
Matching tools generally stop short of this. AWS describes its resolution output as “a consolidated data output table with similar records that have a common match ID” — a grouping and an identifier, not an opinion about which address is right. What the combined record says is a separate step, and the organization owns it.
Four bases, each with a failure mode
Most rules in practice rest on one of four claims about why a value should win.
| Basis | The claim | Where it goes wrong |
|---|---|---|
| Source authority | One system is where this field is legitimately maintained — billing for the invoicing address, HR for employment status | Fails silently when the authoritative system is stale because nobody updates it, or when authority was asserted by an architecture diagram rather than by how people actually work |
| Recency | The most recently updated value is the most current | Confuses a real change with a careless edit, and depends on update timestamps that mean the same thing across systems — a bulk migration can make every row in one source look newest |
| Completeness | A populated value beats a null; a fuller address beats a partial one | Longer is not more correct, and a field can be non-null and wrong. Useful as a tiebreaker, weak as a primary rule |
| Verification status | A value confirmed by a process — an address validated against a postal file, an email that received a click — beats an unconfirmed one | Strongest where available and rarely available for every field; verification also expires, so a confirmation needs a date attached to it |
None is right in general, which is the practical point: a single rule applied to a whole record will be wrong for some of its fields. Rules belong per field. Legal name may come from the system of registration, contact preference from the most recent customer interaction, and credit terms from finance — in the same merged row.
Writing them as ordered fallbacks handles the common case where the preferred source is empty: take the verified value; failing that, the authoritative source’s value; failing that, the most recent non-null. Stating the chain explicitly is what stops the implicit last step — whatever the load order happened to leave — from becoming the rule.
Why these are business decisions
The question “which system is authoritative for a customer’s billing address” has consequences for who receives invoices and who is responsible when one goes astray. It is not settled by knowing anything about the data; it is settled by deciding how the organization works. That makes it the kind of decision that needs a named owner per field, a written record of what was decided, and the reasoning kept beside it.
The failure to avoid is a rule that exists only inside a mapping tool. It still governs what everyone downstream sees, and nobody can find it to argue with it — so when a value looks wrong, the discussion becomes an investigation rather than a decision review. The related quality vocabulary is worth borrowing here: what a survivorship rule is really trying to produce is a record that is accurate, complete, and consistent at once, and those goals can conflict, which is precisely why someone has to choose.
Changing a rule changes history
Because the merged record is derived, editing a rule rewrites outputs that people have already used. A report run last quarter and re-run today can differ for no reason visible in the data. Three habits keep this manageable.
- Never overwrite the sources. The merged value is a derived field; the contributing values must remain, or a rule change becomes unrecoverable and a dispute becomes unanswerable.
- Record which source each field came from. A provenance column per mastered field turns “why does it say this” into a lookup, and it is what makes a bad rule visible in aggregate — one source winning every field is usually a bug in the rule, not a fact about the source.
- Version the rules and date the change. Then a difference between two runs can be attributed, and a reconciliation against the sources can distinguish a rule change from a data change.
Manual overrides deserve a mention of their own. When someone corrects a merged value by hand, that correction is a decision with higher authority than any automated rule, and it has to survive the next run. A pipeline that recomputes the golden record from scratch and silently discards human corrections will be found out by the person who made them.
How survivorship fits with matching, implementation styles, and designing for wrong decisions is worked through in One Customer, Five Systems; the record these rules assemble is a golden record.
References: AWS Entity Resolution User Guide, What is AWS Entity Resolution?; GOV.UK, Meet the data quality dimensions.
Discover more from Insightful Data Lab
Subscribe to get the latest posts sent to your email.