Outlier

An outlier is a point far from the rest of the data. It might be a measurement error, a genuine extreme value, or a sign that two populations have been mixed together — and which of those it is determines what to do about it, so detection is only the first half of the problem.

The two standard rules are the z-score, flagging \(|z| > 3\), and the interquartile fence, flagging anything below \(Q_1 - 1.5 \times IQR\) or above \(Q_3 + 1.5 \times IQR\). They do not agree, and the first one has two failure modes worth knowing before relying on it.

The z-score rule and its two blind spots

import numpy as np

scores = np.array([70, 75, 80, 85, 90, 92, 95, 100, 12], dtype=float)
print(f"  exam scores {scores.astype(int).tolist()}")
print(f"  mean with the 12 included {scores.mean():.4f}, without it {scores[scores > 12].mean():.4f}")
print(f"  median with {np.median(scores):.4f}, without {np.median(scores[scores > 12]):.4f}")

z = (scores - scores.mean()) / scores.std(ddof=0)
print(f"\n  the z-score of that 12 is {z.min():.4f}"
      f" - the rule |z| > 3 does not fire, because with {len(scores)} points"
      f" no z can exceed {np.sqrt(len(scores) - 1):.4f}")

rng = np.random.default_rng(0)
clean = rng.normal(100, 10, size=500)
print(f"\n{'contaminating values':>21}  {'largest |z|':>12}  {'flagged by |z|>3':>17}  {'flagged by the IQR rule':>24}")
for k in (0, 1, 3, 10, 30):
    data = np.concatenate([clean, np.full(k, 400.0)])
    zs = np.abs((data - data.mean()) / data.std(ddof=0))
    q1, q3 = np.percentile(data, [25, 75])
    fence = (data < q1 - 1.5 * (q3 - q1)) | (data > q3 + 1.5 * (q3 - q1))
    print(f"{k:>21}  {zs.max():>12.4f}  {int((zs > 3).sum()):>17}  {int(fence.sum()):>24}")
#   exam scores [70, 75, 80, 85, 90, 92, 95, 100, 12]
#   mean with the 12 included 77.6667, without it 85.8750
#   median with 85.0000, without 87.5000
#
#   the z-score of that 12 is -2.6339 - the rule |z| > 3 does not fire, because with 9 points no z can exceed 2.8284
#
#  contaminating values   largest |z|   flagged by |z|>3   flagged by the IQR rule
#                     0        3.8206                  4                         6
#                     1       17.8407                  1                         7
#                     3       11.8293                  3                         9
#                    10        6.8741                 10                        16
#                    30        4.0420                 30                        35

The exam-score example is the standard one, and it shows the first problem immediately. The 12 is obviously out of place, dragging the mean from 85.875 down to 77.667 while the median barely moves. And its z-score is −2.6339, so the rule \(|z| > 3\) does not fire.

That is not bad luck. With \(n\) points, using the population-standard-deviation convention (ddof=0) above, the largest possible \(|z|\) is \(\sqrt{n-1}\), which for nine points is 2.8284. No value in a nine-point sample can ever have \(|z| > 3\), however extreme it is — the threshold is unreachable below eleven points, and is a weak filter well beyond that.

The second table shows the other failure, which is worse because it worsens as the problem does. One contaminating value at 400 gives a largest \(|z|\) of 17.8407 — caught easily. Thirty of them give 4.0420, and with enough more they would slip under the threshold entirely. The contaminating points inflate the standard deviation that they are then measured against, so a cluster of outliers hides itself. This is called masking, and it is why the mean and standard deviation are poor foundations for a rule meant to find things that distort the mean and standard deviation.

The interquartile rule is less sensitive to the magnitude of extreme values, but contamination can move the quartiles and their fences too. It flags more in every row above, including in the uncontaminated data — six points out of five hundred clean normal draws. Which is a reminder that in a large sample, some points are expected to be far out, and “flagged” does not mean “wrong.”

Deciding what to do

The decision depends on the cause, and the cause is not visible in the number.

• An error — remove or repair it. Repair or remove a value when the source, units, or valid-domain rules establish that it is wrong—for example, a confirmed unit error or a timestamp outside the permitted recording period. Magnitude alone does not establish that a height, income, or historical timestamp is invalid. Correct the originating process where possible.
• A genuine extreme — keep it. Real people are unusually tall and real transactions are unusually large. Removing them because they are inconvenient is deleting the part of the distribution you were asked to model.
• The thing you are looking for — keep it and study it. In fraud, fault detection and rare-disease work the outliers are the positives, and an outlier filter in the pipeline deletes the signal.
• Unclear — use a method that tolerates it. Median and interquartile range instead of mean and standard deviation, absolute error instead of squared, robust regression, or a log transform to compress the tail.

Whichever you choose, do it inside the pipeline rather than to the dataset. A rule fitted on the whole data before splitting uses the test rows’ extremes to define what counts as extreme, and a min–max scaler fitted the same way lets one mistyped value move every other row — the effect the article on normalization measures.

For multivariate data, none of the single-column rules apply: a point can be unremarkable on every feature and still be a strange combination of them. That is what IsolationForest, LocalOutlierFactor and one-class SVMs are for, and their output is a score rather than a verdict — which leaves the same decision above still to be made.

References

• NIST/SEMATECH e-Handbook of Statistical Methods, Detection of Outliers
• scikit-learn User Guide, Novelty and Outlier Detection