Survivorship Bias

Survivorship bias is what happens when an analysis includes only the cases that lasted, and draws conclusions as if those were all the cases. Average tenure computed from current employees omits everyone who left early. Satisfaction measured among active subscribers omits everyone who canceled because they were dissatisfied.

It is related to selection bias and distinct in a way that matters for diagnosis. Selection bias is about who entered the data; survivorship bias is about who left it. The second is easier to miss because the dataset looks complete — every row is real, nothing is malformed, and the absent cases leave no trace.

Data platforms manufacture it

This is not only an analyst’s error. Common and sensible engineering choices produce datasets in which survivors are the only population available.

• Current-state tables. Physical deletion can remove historical membership evidence; filtering only for currently active accounts can exclude former members even when their history remains. If opening and closing timestamps adequately describe membership, rows with opened_at <= t and no closing timestamp or closed_at > t can reconstruct the population at time t. Reopenings and other changes may require a fuller event history.
• Active-user filters. Pipelines routinely restrict to active records for good performance reasons. Any metric computed downstream is then conditioned on survival, and the condition is invisible in the output.
• Deletion requests. Account closure and a request to delete personal data are separate events, and either can occur without the other. When data is deleted under the applicable policy, record the resulting analytical limitation without assuming the missing cases are exactly the closed accounts.

The third is worth sitting with, because it is a genuine tension rather than a mistake. Where applicable law and retention policy permit, suitably aggregated or effectively anonymized departure counts can support the analysis: how many accounts closed, in which month, at what tenure. Assess whether people remain identifiable, including through small groups or linkage with other information. If suitable evidence cannot be retained, state the resulting limit on the analysis.

How it changes a conclusion

The characteristic error is inferring that a behavior causes success when it is actually a marker of having survived. Customers who use an advanced feature show high retention — but customers who were going to stay are also the ones who stayed long enough to discover the feature. The recommendation “drive feature adoption to improve retention” may be right and the data shown does not support it.

Averages move in a predictable direction too. Mean tenure among current customers rises whenever short-lived customers are excluded, so a company with worsening early churn can report improving average tenure. Both numbers are correct, and one of them is answering a question nobody asked.

And forecasts inherit it. Projecting from a surviving population assumes the future resembles that population’s past — predictability depends on “how similar the future is to the past” — while the cases that would have made the past look worse are missing from the record the projection rests on.

Diagnosis and remedies

One question finds it: who has disappeared from this dataset, and is their disappearance related to what I am measuring? If the answer to the second half is yes, the estimate is biased, and collecting more of the same data will make it more precisely biased rather than more accurate.

• Fix the cohort, then measure. Define the population at a point in time — everyone who signed up in March — and follow it forward. A cohort defined at entry makes membership explicit, but detecting later loss still requires checks on membership, outcome availability, and missing observations.
• Keep history, not just state. A table that records when each account opened and closed answers questions a current-state table cannot, and the design must preserve appropriate identifiers, valid time periods, reopenings, and missing-data semantics within the applicable retention policy.
• Record departures explicitly. Treat ending as an event worth capturing — when, at what tenure, and with whatever reason was given. Keep closure and deletion-request events distinct where permitted; an absent row alone does not identify the reason or timing.
• Truncate the window deliberately. When comparing groups with different amounts of elapsed time, compare them over the same number of days from entry. Unequal exposure can otherwise confound the comparison; the direction depends on the outcome, so older cohorts are not always advantaged.

Where none of this is possible — the data is gone, the departures were never recorded — the remaining honest move is to say so. The ASA’s principle that “proper inference requires full reporting and transparency” applies as much to a stated limitation as to a reported number, and an estimate labeled “computed on customers still active, which excludes churned accounts” is usable. The same estimate presented without that clause is not.

How survivorship sits alongside selection and measurement bias, and what else to check before acting on a result, is worked through in Reading a Result Honestly.

References:

• Hyndman & Athanasopoulos, Forecasting: Principles and Practice (3rd ed.), §1.1 What can be forecast?
• Wasserstein & Lazar, The ASA’s Statement on p-Values, The American Statistician 70:129–133, 2016
• the same article in PubMed Central