Machine Learning (ML)

Machine learning fits a rule to examples instead of having someone write the rule by hand. Given inputs \(X\) and outcomes \(y\), the aim is a function that maps one to the other:

\[ y = f(X) + \epsilon \]

where \(\epsilon\) is everything the inputs cannot account for. The families follow from what is available: supervised learning when the outcomes are known, unsupervised when they are not and the aim is structure, semi-supervised when a few are known, reinforcement learning when the signal is a reward from acting, and self-supervised when the data supplies its own targets — predicting the next word from the previous ones, which is how large language models are trained.

The phrase usually attached is “without being explicitly programmed.” That is a fair description of how the rule is obtained and says nothing about whether the rule is any good. The distinction that does is worth making early.

Learning is what survives new data

import numpy as np
from sklearn.datasets import make_regression
from sklearn.model_selection import train_test_split
from sklearn.neighbors import KNeighborsRegressor
from sklearn.linear_model import Ridge

X, y = make_regression(n_samples=1200, n_features=15, n_informative=6, noise=20.0, random_state=0)
xtr, xte, ytr, yte = train_test_split(X, y, test_size=0.4, random_state=0)
"""Two models. One memorizes the training rows exactly; the other fits a rule.
   On the data they were shown, the memorizer looks perfect."""

for name, model in (("1-nearest neighbor (memorizes)", KNeighborsRegressor(1)),
                    ("ridge regression (fits a rule)", Ridge(alpha=1.0)),
                    ("predicting the training mean", None)):
    if model is None:
        on_train = 1 - np.sum((ytr - ytr.mean()) ** 2) / np.sum((ytr - ytr.mean()) ** 2)
        on_test = 1 - np.sum((yte - ytr.mean()) ** 2) / np.sum((yte - yte.mean()) ** 2)
        train_err = np.mean((ytr - ytr.mean()) ** 2)
        test_err = np.mean((yte - ytr.mean()) ** 2)
    else:
        model.fit(xtr, ytr)
        on_train, on_test = model.score(xtr, ytr), model.score(xte, yte)
        train_err = np.mean((ytr - model.predict(xtr)) ** 2)
        test_err = np.mean((yte - model.predict(xte)) ** 2)
    print(f"  {name:<32} R2 on rows it saw {on_train:>7.4f}   on new rows {on_test:>7.4f}"
          f"   error {train_err:>9.1f} -> {test_err:>9.1f}")
#   1-nearest neighbor (memorizes)   R2 on rows it saw  1.0000   on new rows  0.3474   error       0.0 ->   14600.7
#   ridge regression (fits a rule)   R2 on rows it saw  0.9842   on new rows  0.9822   error     372.2 ->     398.3
#   predicting the training mean     R2 on rows it saw  0.0000   on new rows -0.0005   error   23487.2 ->   22382.6

The first model stores the training rows and answers any query with its nearest stored example. On the rows it was given, it is perfect — \(R^2\) of 1.0000, squared error of exactly zero. On rows it has never seen, it manages 0.3474.

The second fits a rule: fifteen weights and an intercept. It is slightly worse on the training rows, 0.9842, and almost identical on new ones, 0.9822. It has learned something about the relationship rather than about the particular rows.

Useful learning is demonstrated by performance on new data against a relevant baseline and task requirement. A small train–test gap alone is not enough: a weak constant predictor can perform equally poorly on both. A model with some gap can still generalize usefully. Splits and leakage checks make the independent performance estimate credible; the gap is one diagnostic rather than the definition of success.

The third row is the other habit worth forming. A rule that ignores the inputs entirely and always predicts the training mean scores 0.0000 on training rows and -0.0005 on this test set. The zero reference for test R² uses the test mean, so a constant learned from the training set need not score exactly zero there. Every model needs comparing against something that stupid, because a number like “0.35” means nothing until you know what zero effort achieves.

The workflow, and where it actually goes wrong

Collect data, clean it, choose a model, train it, evaluate it, deploy it, monitor and retrain. That list is accurate and hides the difficulty in its fifth step.

• How the data was split decides whether the evaluation means anything — see cross-validation, and time-based splits where the data has an order.
• Which metric answers the question the decision asks, since accuracy on imbalanced data can be beaten by predicting nothing.
• Whether any information crossed the split — a scaler fitted on everything, a target-encoded column, a duplicate row — which is the single most common way a project reports a number it cannot reproduce in production.

And the step after deployment is not optional. The rule was fitted to a snapshot; when the world moves, the rule does not move with it, which is why monitoring and retraining appear in the list at all.

What kind of problem you have

Within supervised learning, the type of the target variable decides the task. A number means regression — a house at 1,200 square feet with four rooms predicted at \$300,000. A category means classification — spam or not.

The same algorithm families serve both, so the choice of task is a choice about what you are predicting rather than about which method to use. And as the article on target variables shows, that choice is often not given by the data at all — someone defines it, and the definition changes the problem.

References

• scikit-learn, An introduction to machine learning with scikit-learn
• scikit-learn User Guide, Cross-validation