Anomaly Detection
An anomaly score measures a kind of departure from a reference pattern. Whether that departure matters depends on the application: a sensor fault, a suspicious transaction, and an unusual but valid customer may all look statistically uncommon. Choosing a detector includes choosing features and a reference population. Domain rules, known incidents, targeted inspections, and controlled perturbations can inform that choice even when a complete labeled dataset is unavailable.
What is being detected?
Outlier detection searches for unusual rows within the dataset used to fit the detector, usually assuming most rows describe a useful normal pattern. Novelty detection fits a reference sample and scores new observations. Its reference data should represent normal operation and be sufficiently free of the anomalies it is meant to detect. Neither setup guarantees that statistical rarity identifies the event you care about.
A point can be unusual on its own, unusual only in context, or part of an unusual sequence or group. A reading of 30 may be normal at one operating temperature and abnormal at another. A constant sensor value can look ordinary row by row while its lack of change over time is suspicious. Time-window variation, duplicate rates, or deviations from a context-specific prediction may be better features than raw values alone.
Four scoring rules
Isolation Forest partitions subsamples with random feature splits. Observations isolated in fewer splits tend to receive larger anomaly scores. Local Outlier Factor (LOF) compares a point’s local reachability density with that of its neighbors; it is a relative neighborhood measure, not a test of whether a cluster is globally rare. Its neighborhood size changes what counts as local.
An RBF One-Class SVM fits a kernel-based support boundary. The kernel width and nu affect that boundary; nu is not simply a requested alert fraction. In its standard formulation it bounds the fraction of training margin errors from above and the support-vector fraction from below, subject to numerical fitting behavior. EllipticEnvelope estimates a robust location and covariance and scores squared Mahalanobis distance from the fitted center. Its ellipsoidal, roughly Gaussian reference assumption can be poor for multimodal or strongly non-elliptical data; non-Gaussian data do not automatically make every result useless.
Scaling and feature construction affect distance- and kernel-based methods. Fit preprocessing on the reference/training portion when evaluating new data. Anomaly scores are usually not calibrated probabilities, and different detectors use different scales and sign conventions. In the following code, larger scores always mean more anomalous: scikit-learn scores that increase toward normality are negated.
Three constructed anomaly patterns
The experiment fits and scores each detector on the same contaminated dataset. This is a retrospective outlier-ranking experiment, not an estimate of performance on future unseen data. Synthetic labels identify the generating component only and are used for evaluation. All four methods see the same standardized features within each scenario. Run the Python blocks in order with NumPy, SciPy, and scikit-learn installed.
import numpy as np
from sklearn.ensemble import IsolationForest, RandomForestClassifier
from sklearn.neighbors import LocalOutlierFactor
from sklearn.svm import OneClassSVM
from sklearn.covariance import EllipticEnvelope
from sklearn.preprocessing import StandardScaler
from sklearn.metrics import roc_auc_score, average_precision_score, precision_score, recall_score
def make_anomalies(n=6000, d=10, frac=0.02, kind="shift", seed=0):
if kind not in ("shift", "scatter", "dense"):
raise ValueError("Unknown anomaly pattern")
count = int(n * frac)
if not 0 < count < n:
raise ValueError("Need both normal and anomalous examples")
g = np.random.default_rng(seed)
normal = g.normal(size=(n-count, d))
if kind == "shift":
unusual = g.normal(4.0, 1.0, (count, d))
elif kind == "scatter":
unusual = g.uniform(-6, 6, (count, d))
else:
unusual = g.normal(0.0, 0.05, (count, d))
values = np.vstack([normal, unusual])
labels = np.r_[np.zeros(n-count, dtype=int), np.ones(count, dtype=int)]
order = g.permutation(n)
return values[order], labels[order]
def retrospective_scores(values):
models = {
"IsolationForest": IsolationForest(random_state=0, n_estimators=200),
"OneClassSVM": OneClassSVM(nu=0.05, gamma="scale"),
"EllipticEnvelope": EllipticEnvelope(random_state=0, support_fraction=0.9),
}
scores = {name: -model.fit(values).score_samples(values)
for name, model in models.items()}
lof = LocalOutlierFactor(n_neighbors=20).fit(values)
scores["LOF"] = -lof.negative_outlier_factor_
return scores
experiments = {}
for kind in ("shift", "scatter", "dense"):
values, labels = make_anomalies(kind=kind)
scaled = StandardScaler().fit_transform(values)
scores = retrospective_scores(scaled)
experiments[kind] = (scaled, labels, scores)
for name, score in scores.items():
print(f"{kind:7s} {name:16s} AUC {roc_auc_score(labels, score):.4f}"
f" AP {average_precision_score(labels, score):.4f}")
# shift IsolationForest AUC 1.0000 AP 1.0000
# shift OneClassSVM AUC 0.9498 AP 0.3091
# shift EllipticEnvelope AUC 1.0000 AP 1.0000
# shift LOF AUC 0.3760 AP 0.0150
# scatter IsolationForest AUC 1.0000 AP 0.9999
# scatter OneClassSVM AUC 1.0000 AP 1.0000
# scatter EllipticEnvelope AUC 1.0000 AP 1.0000
# scatter LOF AUC 1.0000 AP 1.0000
# dense IsolationForest AUC 0.0000 AP 0.0102
# dense OneClassSVM AUC 0.0000 AP 0.0102
# dense EllipticEnvelope AUC 0.0000 AP 0.0102
# dense LOF AUC 0.3245 AP 0.0138AUC measures positive-versus-negative ordering, and AP is average precision rather than a trapezoidal PR-curve area. Both require the synthetic labels here. Rounded values of 1.0000 need not mean perfect ordering at full precision. In the scatter case the anomalies are widely spread, which is an easy example for these settings; it is not representative of every anomaly-detection benchmark.
In the shifted cluster, 120 anomalies share the same per-coordinate spread as the normal component but have a different center. Each has at most 119 other anomalous neighbors; LOF uses only twenty neighbors here. Nearby anomalous neighbors can make a point’s relative density look ordinary. This explains a plausible failure mechanism for this construction, not a theorem that LOF cannot detect clustered anomalies. Cluster boundaries, neighborhood size, spacing, and the reference set can change the outcome.
The dense component lies near the normal center with much smaller variance. Under the generator’s labels it is anomalous, but a large-distance or isolation score may place it at the normal end of the ranking. An AUC of zero means every labeled positive scores below every negative if there are no cross-class ties; it is not a claim of Pearson correlation −1. Values displayed as 0.0000 may instead be very small. A detector using temporal variation or group-density changes could express a different anomaly criterion. The rows alone do not establish that a sensor is stuck.
Disagreement is a prompt to investigate
scaled, labels, scores = experiments["shift"]
names = ["IsolationForest", "LOF", "OneClassSVM"]
budget = 120
for i, left in enumerate(names):
for right in names[i+1:]:
a = set(np.argsort(-scores[left], kind="stable")[:budget])
b = set(np.argsort(-scores[right], kind="stable")[:budget])
print(left, "vs", right, "overlap", round(len(a & b) / budget, 4))
# IsolationForest vs LOF overlap 0.0
# IsolationForest vs OneClassSVM overlap 0.35
# LOF vs OneClassSVM overlap 0.55Overlap divides the number of shared flagged rows by the common list size; it is not precision or recall. Two lists can disagree because they emphasize different geometry, or agree while making the same mistake. Scores alone cannot determine which list matches an external event definition. Inspecting known incidents, checking domain constraints, and labeling selected cases provide additional evidence.
Reviewing only alerts estimates their usefulness, not the detector’s recall over all anomalies. Include a representative audit sample from outside the alert list when estimating missed events. Track the sampling probabilities if combining differently selected groups, and use a suitable weighted estimate when reporting population performance. A few labels can be informative without being sufficient for a precise recall estimate.
Score new observations against a reference
For novelty detection, keep reference fitting, threshold selection, and evaluation separate. The next example fits a clean synthetic normal reference, chooses a score cutoff using a separate normal calibration sample, and evaluates new normal and shifted rows. These clean-reference assumptions are stronger than those of the earlier contaminated-data experiment.
g = np.random.default_rng(11)
reference = g.normal(size=(2000, 10))
calibration = g.normal(size=(1000, 10))
future = np.vstack([g.normal(size=(1000, 10)), g.normal(4, 1, size=(40, 10))])
future_labels = np.r_[np.zeros(1000, dtype=int), np.ones(40, dtype=int)]
scale = StandardScaler().fit(reference)
novelty = LocalOutlierFactor(n_neighbors=20, novelty=True).fit(scale.transform(reference))
cal_scores = -novelty.score_samples(scale.transform(calibration))
cutoff = np.quantile(cal_scores, 0.99)
future_scores = -novelty.score_samples(scale.transform(future))
flags = future_scores > cutoff
print("normal false-alert rate", round(flags[future_labels == 0].mean(), 4))
print("shifted recall", round(flags[future_labels == 1].mean(), 4))
print("future AP", round(average_precision_score(future_labels, future_scores), 4))
# normal false-alert rate 0.003
# shifted recall 1.0
# future AP 1.0With LOF’s default outlier mode, training-row scores come from negative_outlier_factor_. With novelty=True, use score_samples for new observations; evaluating the training rows with that method is not equivalent to their original LOF scores. The shifted test points now have a normal reference neighborhood, unlike the fitted anomalous cluster in the earlier example. The two experiments answer different questions.
Here, three of the 1,000 new normal rows trigger alerts, and all forty shifted rows are detected. The 99th percentile targets a roughly 1% upper tail in the calibration distribution; it does not guarantee a 1% future false-alert rate. Finite calibration samples, ties, and distribution change matter. If the calibration sample itself contains anomalies, its upper-tail fraction is not a normal-only false-positive rate. Choose thresholds using justified costs, audited performance, or a review budget, and monitor alert rates under changing traffic. Scores are not anomaly probabilities without further modeling and validation.
For scikit-learn IsolationForest with fixed data, seed, and other parameters, numeric contamination determines a training-score quantile offset used by predict; it does not change the fitted trees. Ties and quantile interpolation mean it need not flag exactly c times n rows, and future alert fractions need not equal c. The auto setting uses the implementation’s fixed offset, not an estimate of the true anomaly prevalence. These statements do not apply to One-Class SVM’s nu, which affects fitting. Operating-point metrics are discussed in Classification Metrics and Thresholds.
Use labels without hiding the selection process
A detector can help prioritize labeling even when its own ranking is not strong enough for deployment. Targeted sampling may find more positives per reviewed case, allowing a supervised model to learn a useful distinction. It may also underrepresent other anomaly types and normal regions. Compare labeling strategies on a separate evaluation sample, keeping preprocessing and model selection away from that sample.
A uniform random audit stream can estimate population quantities when retained as such. Pooling it with targeted labels and reporting an unweighted mean is generally biased. For training, exploration can broaden coverage, but it does not guarantee that all blind spots disappear. Combine label efficiency with evaluation of missed cases and important subgroups. The broader setup is covered in Semi-Supervised and Active Learning.
Exercises
1. Contamination changes the cutoff. Refit IsolationForest with three contamination settings while keeping its seed and tree parameters fixed. Check raw scores as well as flagged counts. Then test identical rows.
Solution
scaled, labels, _ = experiments["shift"]
raw = None
for contamination in (0.001, 0.02, 0.1):
model = IsolationForest(random_state=0, n_estimators=200,
contamination=contamination).fit(scaled)
score = model.score_samples(scaled)
if raw is None:
raw = score
flags = model.predict(scaled) == -1
print("c", contamination, "same scores", np.allclose(score, raw),
"flagged", int(flags.sum()),
"precision", round(precision_score(labels, flags, zero_division=0), 4),
"recall", round(recall_score(labels, flags), 4))
tied = np.ones((50, 2))
model = IsolationForest(random_state=0, n_estimators=20, contamination=0.2).fit(tied)
print("identical rows flagged", int((model.predict(tied) == -1).sum()))
# c 0.001 same scores True flagged 6 precision 1.0 recall 0.05
# c 0.02 same scores True flagged 120 precision 1.0 recall 1.0
# c 0.1 same scores True flagged 600 precision 0.2 recall 1.0
# identical rows flagged 0The fitted scores agree across contamination settings, while the cutoff changes. On identical rows the scores tie at the cutoff, and the strict outlier decision does not force ten of the fifty rows to be flagged. To inspect exactly a fixed number of cases, use an explicit top-k policy with a documented tie rule.
The true anomaly rate does not automatically identify the optimal threshold. The appropriate operating point depends on costs, capacity, and the score distributions. For a fixed ranking, lowering the cutoff cannot decrease recall, but precision need not decrease monotonically at every threshold. Once scores are available, a cutoff sweep avoids refitting the same forest.
2. Label the top of a ranking or sample at random. Compare supervised models trained from random and detector-selected labels. Retain random samples containing only one class in the reported averages.
Solution
from sklearn.model_selection import train_test_split
g = np.random.default_rng(0)
values = g.normal(size=(20000, 20))
labels = np.zeros(20000, dtype=int)
labels[:400] = 1
values[:400, :3] += 2
order = g.permutation(len(labels))
values, labels = values[order], labels[order]
Xtrain, Xtest, ytrain, ytest = train_test_split(
values, labels, test_size=0.5, random_state=0, stratify=labels)
scaler = StandardScaler().fit(Xtrain)
Ztrain, Ztest = scaler.transform(Xtrain), scaler.transform(Xtest)
iso = IsolationForest(random_state=0, n_estimators=300).fit(Ztrain)
train_scores, test_scores = -iso.score_samples(Ztrain), -iso.score_samples(Ztest)
print("unsupervised AP", round(average_precision_score(ytest, test_scores), 4))
def fit_from_labels(indices):
observed = ytrain[indices]
if len(np.unique(observed)) == 1:
return np.full(len(ytest), observed.mean()), True
clf = RandomForestClassifier(n_estimators=200, random_state=0, n_jobs=1)
clf.fit(Ztrain[indices], observed)
return clf.predict_proba(Ztest)[:, 1], False
for budget in (50, 100, 300, 1000):
random_ap, random_positive, single_class = [], [], 0
for repeat in range(8):
indices = np.random.default_rng(repeat).choice(len(ytrain), budget, replace=False)
prediction, fallback = fit_from_labels(indices)
random_ap.append(average_precision_score(ytest, prediction))
random_positive.append(ytrain[indices].sum())
single_class += int(fallback)
selected = np.argsort(-train_scores, kind="stable")[:budget]
prediction, fallback = fit_from_labels(selected)
print("budget", budget, "random AP", round(np.mean(random_ap), 4),
"random positives", round(np.mean(random_positive), 2), "single-class runs", single_class,
"targeted AP", round(average_precision_score(ytest, prediction), 4),
"targeted positives", int(ytrain[selected].sum()), "targeted fallback", fallback)
# unsupervised AP 0.1
# budget 50 random AP 0.2486 random positives 1.38 single-class runs 1 targeted AP 0.7222 targeted positives 15 targeted fallback False
# budget 100 random AP 0.2817 random positives 1.12 single-class runs 4 targeted AP 0.7792 targeted positives 24 targeted fallback False
# budget 300 random AP 0.5983 random positives 5.12 single-class runs 0 targeted AP 0.8074 targeted positives 52 targeted fallback False
# budget 1000 random AP 0.7117 random positives 19.88 single-class runs 0 targeted AP 0.8164 targeted positives 96 targeted fallback FalseThe labeled anomalies shift only three of twenty coordinates. The remaining coordinates can distract an isolation-based score, but this experiment alone does not identify the exact cause of its performance. Isolation Forest is not a density estimator. At a budget of fifty labels, targeted sampling finds fifteen positives and reaches AP 0.7222; random sampling averages 1.38 positives and AP 0.2486 across eight draws. This construction gives targeted labeling a useful starting set despite the detector’s test AP of 0.1000.
A supervised classifier can fit even with one positive, although its ability to generalize from that example is uncertain. With only one observed class, this comparison explicitly uses a constant-score fallback. Its AP equals test prevalence because all test scores tie. Those runs remain in the mean; excluding them would condition the random-sampling result on having found both classes.
Random sampling is repeated eight times; targeted selection and classifier randomness are fixed. Both use the same held-out test set. This does not establish a universal label-efficiency ratio or an uncertainty interval over datasets. Targeted labels can miss anomaly types outside the detector’s favored region, so evaluate beyond the selected region and consider exploration when gathering additional labels.
3. Rank consensus. Average normalized ranks from three detectors on the existing shift and dense scenarios. Does this validate the anomaly definition?
Solution
from scipy.stats import rankdata
for kind in ("shift", "dense"):
_, labels, all_scores = experiments[kind]
selected = {name: all_scores[name] for name in ("IsolationForest", "LOF", "OneClassSVM")}
ranks = [(rankdata(score, method="average") - 1) / (len(score) - 1)
for score in selected.values()]
consensus = np.mean(ranks, axis=0)
print(kind, "member AUCs", [round(roc_auc_score(labels, score), 4)
for score in selected.values()],
"consensus AUC", round(roc_auc_score(labels, consensus), 4))
# shift member AUCs [1.0, 0.376, 0.9498] consensus AUC 0.7836
# dense member AUCs [0.0, 0.3245, 0.0] consensus AUC 0.0879Average ranks put detectors on a common ordinal scale and assign equal ranks to ties. This discards score gaps. The consensus can outperform or underperform a member; its AUC is not mathematically constrained to lie between all member AUCs. In the shifted case, consensus reaches AUC 0.7836, below Isolation Forest’s 1.0000; in the dense case it reaches only 0.0879. Combining scores has not resolved the mismatch between the dense-component labels and these scoring rules.
On the central dense component, the methods may rank the labeled anomalies near the normal end, but that does not mean they agree perfectly on every row or have identical assumptions. Agreement and seed stability are diagnostics, not labels. They can reveal sensitivity and inconsistency without proving that a shared criterion matches the desired event.
Known incidents, domain rules, and realistic injected anomalies can test that criterion. An injected example validates detection of that constructed mechanism, not every unknown anomaly type. Use rank aggregation only as a candidate procedure to evaluate, rather than treating consensus as correctness.
References
- scikit-learn: novelty and outlier detection.
- scikit-learn: LocalOutlierFactor.
- scikit-learn: IsolationForest.
- scikit-learn: OneClassSVM.
Discover more from Insightful Data Lab
Subscribe to get the latest posts sent to your email.