Data Preparation for Machine Learning

Filling a missing value, encoding a category, and clipping an extreme observation change the inputs a model receives. This article follows those choices through small experiments: what information each transformation retains, what assumptions it introduces, and how to evaluate it without using held-out data to fit the transformation.

Missing values are data

A missing value can mean that a measurement failed, a question was skipped, or a field was not applicable. Investigate how it was produced before choosing a replacement. In the constructed example below, the measured values are independent of the target, but positive cases are more likely to have a missing measurement.

The examples use NumPy and scikit-learn. cross_val_score(..., cv=5) reports accuracy on five held-out folds for these classifiers. A pipeline fits its imputer only on each training fold; add_indicator=True appends a missingness column for features with missing values during that fit. The original input has one column, so the flagged version has two here.

import numpy as np
from sklearn.impute import SimpleImputer
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score
from sklearn.pipeline import make_pipeline
from sklearn.dummy import DummyClassifier

rng = np.random.default_rng(0)
n = 4000
y = rng.integers(0, 2, n)
feature = rng.normal(size=n)
missing = rng.random(n) < np.where(y == 1, 0.55, 0.10)
observed = feature.copy()
observed[missing] = np.nan
X = observed.reshape(-1, 1)

for flag in (False, True):
    model = make_pipeline(SimpleImputer(strategy="mean", add_indicator=flag),
                          LogisticRegression())
    name = "impute + missing flag" if flag else "impute only"
    print(f"{name:24s} {cross_val_score(model, X, y, cv=5).mean():.4f}")
print(f"majority baseline        {cross_val_score(DummyClassifier(strategy='most_frequent'), X, y, cv=5).mean():.4f}")
print(f"missing rate: positives {missing[y == 1].mean():.3f}"
      f"  negatives {missing[y == 0].mean():.3f}")
# impute only              0.5485
# impute + missing flag    0.7213
# majority baseline        0.5072
# missing rate: positives 0.540  negatives 0.092

Accuracy is 0.5485 with imputation alone and 0.7213 with the flag, compared with a majority baseline of 0.5072. The original feature values carry no target signal in this generating process. A missingness flag is 1 when a value is absent and 0 when it is present. Adding that column gives logistic regression a separate coefficient for missingness. Filling gaps with a constant can leave a recognizable pattern, but does not give this linear model an explicit indicator for it.

With one numeric input, logistic regression fits a linear score and a monotone probability curve. It cannot isolate an exact interior constant as a separate category. A tree might detect the concentrated imputed values through nearby thresholds. The impute-only score here should therefore not be explained as logistic regression detecting equality to the imputation constant; the printed majority baseline also gives a comparison for the sample’s class imbalance.

A missingness flag is a candidate feature to evaluate, especially when the collection process is informative. Check that the missingness is observable at prediction time and that the relationship persists in the intended setting. For example, a hospital changing when it orders a test can change the meaning of an absent result. An indicator need not improve every model, particularly one that already handles missing values.

What imputation costs

Replacing missing values by a mean puts all those rows at one point. Under missingness completely at random, replacing a fraction of one variable by its population mean reduces its variance and attenuates its Pearson correlation with a fully observed partner. Other missingness patterns need not have the same effect. The following descriptive experiment compares the original and filled arrays; it does not evaluate a fitted predictor.

import numpy as np

rng = np.random.default_rng(1)
z = rng.normal(50, 10, 5000)
partner = 0.8 * z + 0.6 * rng.normal(0, 10, 5000)

dropped = rng.random(5000) < 0.30
with_gaps = z.copy(); with_gaps[dropped] = np.nan
filled = np.where(np.isnan(with_gaps), np.nanmean(with_gaps), with_gaps)

print(f"sd  original {z.std():.4f}   observed only {np.nanstd(with_gaps):.4f}"
      f"   mean-imputed {filled.std():.4f}")
print(f"corr  complete rows {np.corrcoef(z[~dropped], partner[~dropped])[0, 1]:.4f}"
      f"   mean-imputed {np.corrcoef(filled, partner)[0, 1]:.4f}")
# sd  original 10.0064   observed only 10.1205   mean-imputed 8.4030
# corr  complete rows 0.7972   mean-imputed 0.6687

Here each value is removed independently with probability 0.30: missing completely at random (MCAR). The standard deviation falls from 10.01 to 8.40 and the correlation from 0.80 to 0.67. With an observation probability \(r\) and population-mean replacement, variance becomes \(r\operatorname{Var}(Z)\), and correlation with a fully observed partner is multiplied by \(\sqrt r\). For \(r=0.70\), that factor is about 0.837. The code uses a finite sample and its observed mean, so its values need not match that population calculation exactly. Treating filled values as ordinary measurements can mislead covariance analysis and uncertainty estimates.

Model-based imputation predicts a missing value from other observed columns. It can use relationships that a column mean ignores, but a deterministic prediction still does not restore the full missing-value uncertainty. Multiple imputation uses several plausible completed datasets and appropriate pooling to represent that uncertainty under assumptions about missingness and the imputation model; it is not automatically correct for every problem. For prediction, compare practical alternatives inside the training procedure. Some tree implementations, including histogram gradient boosting, can learn how to route missing values directly. Native handling also needs validation against the deployment setting.

Categorical encoding

One-hot encoding gives each category its own indicator column. With categories a, b, and c, the value b becomes [0, 1, 0]. This avoids assigning numerical distances between category labels, but many levels can produce a large feature matrix and leave little training data for rare categories.

A held-out or future case may contain a category that was absent when the encoder was fitted. The next example makes that case explicit.

import numpy as np
from sklearn.preprocessing import OneHotEncoder

train = np.array([["a"], ["b"], ["c"], ["a"], ["b"]])
test = np.array([["a"], ["d"]])                # "d" was never seen

for handle in ("error", "ignore"):
    enc = OneHotEncoder(handle_unknown=handle, sparse_output=False).fit(train)
    try:
        print(f"handle_unknown={handle:7s} -> {enc.transform(test).tolist()}")
    except ValueError as e:
        print(f"handle_unknown={handle:7s} -> ValueError: {str(e)[:52]}")
# handle_unknown=error   -> ValueError: Found unknown categories ['d'] in column 0 during tr
# handle_unknown=ignore  -> [[1.0, 0.0, 0.0], [0.0, 0.0, 0.0]]

Raising an error is useful when an unseen category signals malformed input. With handle_unknown="ignore", an unknown category contributes zeros to this field’s encoded columns; a linear model then uses its intercept and other features. That fallback has no category-specific estimate learned from training. Missing values are a separate case: OneHotEncoder can learn a category for NaN when it appears during fitting. If it is unseen, the unknown-category policy applies. With a dropped reference category, an all-zero encoding can also coincide with that reference. Test the exact combination of missing-value handling, category dropping, and unknown-category handling you intend to use.

Fit the encoder on training rows and test its behavior on held-out levels, missing entries, and rare levels. Offline evaluation can expose these failures if its splits contain unseen categories. Fitting the encoder on the whole dataset first hides that particular test and uses information from the evaluation rows.

Scaling

Standardization uses \(z=(x-\mu)/s\), with the mean \(\mu\) and standard deviation \(s\) fitted on training rows. If those are 100 and 20, a value of 140 becomes 2. Min-max scaling maps the training extrema to a chosen range; new values can fall outside it unless clipped. Robust scaling centers by the median and divides by the interquartile range (75th minus 25th percentile). Its fitted scale is less sensitive to extremes, but it does not remove them. The choice depends on the model, the units, and which differences should count as large.

• Distance-based methods — k-NN, k-means, SVM with an RBF kernel — are sensitive to feature scales. A large numeric scale can dominate distances. Standardization is useful when the units should not determine feature weights, but domain-specific weights or already comparable units can justify another choice.
• Regularized linear models are sensitive to units because a common coefficient penalty changes its effect when a feature is rescaled. Standardization is one way to make that choice explicit; it is not mathematically required, and different feature penalties may be intentional.
• Gradient-based optimization often benefits from better conditioning, as explained in Optimization for Machine Learning. Scaling alone does not resolve every source of poor conditioning.
• Tree-based models usually do not need standardization. Positive affine rescaling preserves the candidate training partitions of an ideal axis-aligned threshold tree. Rounding, approximate split searches, and ties can affect implementations; a transformation that merges distinct values can also change the available splits.

Fit data-dependent transformations on training rows and reuse them on evaluation rows and future inputs. Within cross-validation, fit them separately inside each training fold. A Pipeline combines preprocessing with the estimator so that cross-validation fits both in the right place. This applies to imputation and encoding as well as scaling; Train, Validation, Test, and Data Leakage explains the evaluation design.

The same question on real data

The Adult census dataset lets us inspect missingness without designing its relationship to the target. The loader below may download data on its first run. Missing entries can arrive as NaN or as a literal question mark depending on the loading path, so the code checks both. This is a descriptive audit of the dataset, not a test of predictive improvement from adding a flag.

import numpy as np
from fairlearn.datasets import fetch_adult

d = fetch_adult(as_frame=True)
X = d.data.copy()
y = (d.target == ">50K").astype(int)

miss = X.isin(["?"]) | X.isna()
cols = miss.sum()[lambda s: s > 0].sort_values(ascending=False)
print(f"rows {len(X):,}  columns {X.shape[1]}")
for c, n in cols.items():
    print(f"  {c:16s} missing {n:5,} ({n / len(X) * 100:5.2f}%)")

anymiss = miss.any(axis=1)
print(f"rows with any missing value: {anymiss.sum():,} ({anymiss.mean() * 100:.2f}%)")
print(f">50K rate, complete rows {y[~anymiss].mean():.4f}"
      f"   rows with missing {y[anymiss].mean():.4f}")
# rows 48,842  columns 14
#   occupation       missing 2,809 ( 5.75%)
#   workclass        missing 2,799 ( 5.73%)
#   native-country   missing   857 ( 1.75%)
# rows with any missing value: 3,620 (7.41%)
# >50K rate, complete rows 0.2478   rows with missing 0.1323

Missing entries occur in three columns and affect 7.41% of rows. The observed rate of income above the threshold is 24.78% among complete rows and 13.23% among rows with at least one missing entry. This shows an association between completeness and the target in this dataset. It does not compare the predictive value of missingness with that of the other features.

Deleting all affected rows changes the population represented in training: incomplete cases have a different observed outcome rate. If deployment includes them, a model trained only on complete cases needs evaluation on that intended population. An imputer with indicators or a model with native missing-value handling can retain those rows, but its performance must be measured. Also, dropna() removes actual missing entries; a literal ? must first be converted to a missing marker or handled by an explicit mask.

Exercises

1. Which models care about scale. Give two independent features equal roles in the target rule but numeric scales differing by a factor of 10,000, and compare k-NN, logistic regression, and a random forest with and without standardization.

Compare the size of the score changes. Does the experiment establish why each change occurred?

import numpy as np
from sklearn.model_selection import cross_val_score
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.neighbors import KNeighborsClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import RandomForestClassifier

rng = np.random.default_rng(0)
n = 1500
x1 = rng.normal(0, 1, n)
x2 = rng.normal(0, 1, n) * 10_000              # same standardized distribution as x1
y = (x1 + x2 / 10_000 > 0).astype(int)
X = np.column_stack([x1, x2])

for name, make in (("kNN", lambda: KNeighborsClassifier(5)),
                   ("logistic", lambda: LogisticRegression(max_iter=5000)),
                   ("random forest",
                    lambda: RandomForestClassifier(n_estimators=100, random_state=0))):
    raw = cross_val_score(make(), X, y, cv=5).mean()
    scaled = cross_val_score(Pipeline([("s", StandardScaler()), ("m", make())]),
                             X, y, cv=5).mean()
    print(f"{name:14s} raw {raw:.4f}  scaled {scaled:.4f}  change {scaled - raw:+.4f}")
# kNN            raw 0.7213  scaled 0.9833  change +0.2620
# logistic       raw 0.9927  scaled 0.9973  change +0.0047
# random forest  raw 0.9807  scaled 0.9807  change +0.0000

2. Trimming has to reach the outliers. Corrupt 1.6% of rows with high-leverage points: inputs far from the bulk of the data that can strongly influence a fitted slope. Try winsorizing and dropping using the 1st/99th, 2nd/98th, and 5th/95th percentile bounds. Report the recovered slope in each case, against a Huber fit with no trimming at all.

Check which rows each cutoff actually changes. Ties at a percentile boundary matter.

import numpy as np
from sklearn.linear_model import Ridge
from scipy.optimize import minimize_scalar

rng = np.random.default_rng(1)
n = 500
x = rng.normal(size=n)
y = 2 * x + 0.4 * rng.normal(size=n)
x[:8], y[:8] = 8.0, rng.normal(0, 1, 8)        # 8 of 500 = 1.6%, at high leverage

slope = lambda X, Y: float(Ridge(alpha=1e-9).fit(X.reshape(-1, 1), Y).coef_[0])
print(f"true slope        2.0000")
print(f"keep as is        {slope(x, y):.4f}")
for q in (1, 2, 5):
    lo, hi = np.percentile(x, [q, 100 - q])
    keep = (x >= lo) & (x <= hi)
    print(f"winsorize {q}%      {slope(np.clip(x, lo, hi), y):.4f}"
          f"    drop {q}%  {slope(x[keep], y[keep]):.4f}   (kept {keep.sum()})")

huber = lambda r, d=1.0: np.where(np.abs(r) <= d, 0.5 * r ** 2, d * (np.abs(r) - 0.5 * d))
h = minimize_scalar(lambda b: np.mean(huber(y - b * x)), bounds=(-5, 5),
                    method="bounded").x
print(f"huber, no trimming       {h:.4f}")
# true slope        2.0000
# keep as is        0.8892
# winsorize 1%      0.8889    drop 1%  0.8527   (kept 495)
# winsorize 2%      1.8046    drop 2%  2.0012   (kept 480)
# winsorize 5%      2.0051    drop 5%  1.9927   (kept 450)
# huber, no trimming       1.8401

3. Encoding a 150-level category. Compare one-hot, ordinal, and frequency encoding on a categorical column whose levels genuinely differ in their effect on the target.

Compare each encoding with a majority classifier trained on the same folds. Explain which relationships a one-coefficient logistic model can express.

The loop makes each training/evaluation split explicit. tr and te hold row indices; np.bincount counts training occurrences of each code. Adding None as an index turns a vector into the one-column matrix the estimator expects. The category codes 0 through 149 are fixed identifiers from this synthetic setup, not an order learned from the outcomes.

import numpy as np
from sklearn.preprocessing import OneHotEncoder
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import StratifiedKFold
from sklearn.dummy import DummyClassifier

rng = np.random.default_rng(2)
n, K = 3000, 150
cat = rng.integers(0, K, n)
effect = rng.normal(size=K)
y = (effect[cat] + 0.5 * rng.normal(size=n) > 0).astype(int)

scores = {name: [] for name in ("one-hot", "ordinal", "frequency", "majority baseline")}
for tr, te in StratifiedKFold(5).split(cat, y):
    enc = OneHotEncoder(sparse_output=False, handle_unknown="ignore")
    hot_tr = enc.fit_transform(cat[tr, None])
    hot_te = enc.transform(cat[te, None])
    counts = np.bincount(cat[tr], minlength=K)
    pairs = {
        "one-hot": (hot_tr, hot_te),
        "ordinal": (cat[tr, None].astype(float), cat[te, None].astype(float)),
        "frequency": (counts[cat[tr], None].astype(float),
                      counts[cat[te], None].astype(float)),
    }
    for name, (train_X, test_X) in pairs.items():
        model = LogisticRegression(max_iter=2000).fit(train_X, y[tr])
        scores[name].append(model.score(test_X, y[te]))
    base = DummyClassifier(strategy="most_frequent").fit(hot_tr, y[tr])
    scores["majority baseline"].append(base.score(hot_te, y[te]))

for name, values in scores.items():
    print(f"{name:20s} {np.mean(values):.4f}")
# one-hot              0.8213
# ordinal              0.4943
# frequency            0.5123
# majority baseline    0.5023