Machine Learning Fundamentals: Problems, Data, and Workflow

Machine learning fits a function from data instead of writing it by hand. Fitting a model to a sample does not by itself show how well it will perform on new data, and much of what follows in this curriculum exists to close that gap. This article introduces the vocabulary, the problem types, and the workflow, then uses examples to examine memorization, baselines, and changing feature relationships.

The vocabulary

In the tabular examples here, an observation is one row. Its features are the columns used as input, written \(x\), and in a supervised task, its label or target is the outcome being predicted, written \(y\). A model is a function \(f_\theta\) with parameters \(\theta\); training chooses \(\theta\) from data; inference applies \(f_\theta\) to new rows. Here inference means running the trained model; statistical inference, discussed earlier, concerns what data allow us to conclude about a population or process. Hyperparameters are the settings fixed outside the parameter-fitting procedure — model capacity, regularization strength, learning rate. They are not chosen by the fit itself, but they can be chosen from data: the usual method is to try several values and keep the one with the best validation performance.

Evaluation depends on how the observations were collected and where predictions will be used. Many of the standard evaluation methods in this curriculum assume the rows are independent draws from a fixed distribution, and that deployment data comes from that same distribution. Dependence between observations, or a change in the distribution, can make an evaluation misleading even when nothing else is wrong. Neither assumption is unavoidable — time series methods model the dependence explicitly — but both are worth checking before trusting a number.

Problem types

The question depends on the decision you need to make and the information available. Predicting a duration is regression; predicting whether it exceeds ten minutes is classification; ordering jobs by predicted duration is ranking. These tasks can sometimes use the same model: a regression output can be thresholded or sorted. They still require different evaluation criteria. The exercises compare separate training approaches with different uses of a single fitted model.

Fitting training labels does not establish generalization

A flexible model can fit a training sample without finding a relationship that holds on new data. Given distinct inputs, a model with enough capacity can reproduce the training labels exactly — including labels generated independently of the inputs. (Identical inputs with conflicting labels are the exception: no deterministic function can match both.)

The demonstration below uses k-nearest neighbors, which uses distances to stored training points: to predict the label of a new point, find the \(k\) training points closest to it and take the majority of their labels. At \(k=1\) that is a single neighbor. Here distance is Euclidean distance, with all generated features on the same scale. Two scikit-learn methods appear throughout this series: .fit(X, y) trains a model on the given rows, and .predict(X) returns its predictions for rows you hand it.

The examples require NumPy and scikit-learn and use rows for observations. X has shape (400, 20): 400 observations with 20 features each. y contains 400 binary labels. train_test_split places half the rows in training and half in testing, and accuracy_score computes the fraction of labels predicted correctly. These simulations compare predetermined settings to illustrate behavior; using the test results to choose a setting would require a separate final evaluation.

import numpy as np
from sklearn.neighbors import KNeighborsClassifier
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score

rng = np.random.default_rng(0)
X = rng.normal(size=(400, 20))
y = rng.integers(0, 2, 400)                    # labels independent of X
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.5, random_state=0)

for k in (1, 5, 25):
    m = KNeighborsClassifier(k).fit(X_tr, y_tr)
    print(f"k={k:2d}  train {accuracy_score(y_tr, m.predict(X_tr)):.3f}"
          f"  test {accuracy_score(y_te, m.predict(X_te)):.3f}")
# k= 1  train 1.000  test 0.545
# k= 5  train 0.730  test 0.500
# k=25  train 0.600  test 0.420

There is no population relationship between \(X\) and \(y\) in this generating process, although a finite sample can contain accidental associations. The 1-nearest-neighbor model still achieves perfect training accuracy, because every training point is its own nearest neighbor. Test accuracy is 0.545; the expected accuracy under this generating process is 0.5, and 0.545 is within ordinary sampling variation of it. For 200 independent fair labels, the accuracy’s standard deviation is \(\sqrt{0.5(1-0.5)/200}\approx0.035\), so 0.545 is about 1.27 standard deviations above 0.5. Changing \(k\) does not create a predictive relationship in these labels.

Perfect training accuracy therefore does not establish predictive performance on new data. Training scores still help diagnose fit. Distinguishing limited capacity from incomplete optimization also requires examining the training objective and the optimization process, since accuracy can sit still while the loss and the parameters keep moving. To support an empirical claim about performance on new data, evaluate on data that did not drive fitting or model selection.

Compare against a simple baseline

A useful first comparison is a prediction that ignores the features. Which constant that is depends on the metric: for accuracy it is the majority class, for squared error the mean, for absolute error the median. All three are computed on the training data and then applied unchanged to the evaluation split.

import numpy as np
from sklearn.dummy import DummyClassifier
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score

rng = np.random.default_rng(1)
n = 2000
y = (rng.random(n) < 0.08).astype(int)         # probability 0.08 of a positive label
X = rng.normal(size=(n, 5))
X_tr, X_te, y_tr, y_te = train_test_split(
    X, y, test_size=0.3, random_state=0, stratify=y)

d = DummyClassifier(strategy="most_frequent").fit(X_tr, y_tr)
print(f"majority-class accuracy {accuracy_score(y_te, d.predict(X_te)):.4f}")
print(f"positives in the test set {y_te.mean():.4f}")
# majority-class accuracy 0.9233
# positives in the test set 0.0767

The split uses stratify=y to keep class proportions approximately the same in training and testing. The generating probability is 8%, while the realized positive fraction in the test set is 7.67%. A model that always predicts “no” is therefore 92.3% accurate here and identifies none of the positive cases. An accuracy figure reported without its baseline is hard to judge: on imbalanced problems the baseline is usually high enough that a model can post an impressive-looking number while finding nothing the task cares about.

Other useful comparisons depend on the task: the last observed value for forecasting, or the current rule-based system for a replacement project. Compare predictions on the same eligible cases with the same metric and information available at prediction time. Human annotators’ agreement can help describe label ambiguity, but it is not automatically a model-performance ceiling or a directly comparable baseline.

The workflow

The steps below refer to three splits, and the difference between them is what makes the rest work. The training rows fit the model’s parameters. The validation rows guide the choice between models and hyperparameter settings. Repeatedly adapting those choices to the same validation results can overfit that split, creating a risk that its score overstates performance on new data. The test rows provide the final evaluation after model choices are fixed. Inspecting that result and then revising the model uses the test set for selection; another independent evaluation is needed for the revised procedure. Anything fitted to data — a scaler’s mean and standard deviation, a category encoding, an imputation value — is computed on the training rows alone and then applied unchanged to the other two. Fitting a scaler on all the rows and then splitting lets the validation and test rows influence the numbers they are later evaluated with.

• Frame the decision. What action will the prediction change, and what does an error cost in each direction?
• Assemble the data as it will exist at inference time, not as it exists in a warehouse after the fact.
• Design the split before fitting preprocessing. Keep related entities or duplicates together where needed, and use chronological splits when future prediction is the goal. Learn preprocessing statistics from training rows. Fixed conversions, such as changing a known unit, do not themselves learn from held-out data.
• Establish a baseline and the metric that reflects the decision.
• Fit a simple model and read its errors before reaching for a complex one.
• Tune with validation data, never with the test set.
• Evaluate once on untouched data and report the uncertainty alongside the number.
• Deploy the whole pipeline, including preprocessing, and monitor the inputs as well as the outputs.

The separation protects the final evaluation from choices guided by its outcomes. A scikit-learn Pipeline helps keep learned preprocessing with the estimator and refits it inside each training fold during cross-validation. If tuning is followed by refitting on the combined training and validation data, preprocessing is refitted there too; the final test set remains separate. Deployment must use the same feature definitions, ordering, and transformations as the evaluated model.

What machine learning does not do

Predictive fitting alone does not establish causation. A model that predicts churn from support-ticket volume has learned an association; that association does not establish that suppressing tickets would reduce churn, nor that it would not. A causal claim needs an appropriate study design and identifying assumptions; predictive accuracy alone does not supply them.

Extrapolation beyond the observed range needs assumptions that the training score cannot check. A tree with constant leaves is piecewise constant, so, with the other features held fixed, moving one feature beyond its training range eventually stops changing the prediction ; if other features also change, the point can enter a different leaf and receive a different prediction. A linear regression extends its fitted affine relationship outside that range; whether it remains useful depends on the process being predicted. And it does not invent information: if the label remains uncertain once every available feature is known, perfect prediction is impossible. In classification, the lowest expected misclassification rate achievable with those features is the Bayes error rate under 0–1 loss. It implies a maximum achievable expected accuracy of one minus that rate, though a fitted model need not attain it.

Exercises

The first exercise uses Ridge, a linear regression fitted with a penalty on squared coefficient sizes, and LogisticRegression, a classifier despite its name. MAE is mean absolute error: average the absolute differences between predicted and observed durations, in minutes here. The threshold is computed from training durations and held fixed for evaluation.

1. One dataset, three framings. On the log-normal duration target generated below, (a) fit a regression, (b) fit a classifier for “above the median”, and (c) threshold the regression’s output at the median. Report the accuracy of (b) against (c) and the MAE of (a) against a log-target regression.

You should get, on this generated data: a direct classifier that beats the thresholded regression, and a log-target fit that beats the raw one.

import numpy as np
from sklearn.linear_model import LogisticRegression, Ridge
from sklearn.metrics import accuracy_score, mean_absolute_error

rng = np.random.default_rng(0)
n = 4000
x = rng.normal(size=(n, 3))
wait = np.exp(1.0 + 0.5 * x[:, 0] + 0.3 * rng.normal(size=n))
tr, te = slice(0, 3000), slice(3000, None)
thr = np.median(wait[tr])
late = (wait > thr).astype(int)

pred = Ridge().fit(x[tr], wait[tr]).predict(x[te])
clf = LogisticRegression().fit(x[tr], late[tr])
log_pred = np.exp(Ridge().fit(x[tr], np.log(wait[tr])).predict(x[te]))

print(f"threshold {thr:.3f} min, base rate {late[te].mean():.4f}")
print(f"regression MAE                  {mean_absolute_error(wait[te], pred):.3f}")
print(f"log-target regression MAE       {mean_absolute_error(wait[te], log_pred):.3f}")
print(f"direct classifier accuracy      {accuracy_score(late[te], clf.predict(x[te])):.4f}")
print(f"regression thresholded accuracy {accuracy_score(late[te], (pred > thr).astype(int)):.4f}")
# threshold 2.723 min, base rate 0.4890
# regression MAE                  0.877
# log-target regression MAE       0.709
# direct classifier accuracy      0.8180
# regression thresholded accuracy 0.7920

2. A predictive feature that makes the model worse. Train with a feature that matches the label with probability 0.95, then evaluate on the held-out rows when that link is broken and when it is reversed. Compare against a model that never saw the feature.

You should get: a model without the feature that beats the model with it, under the simulated changes in the feature–label relationship.

import numpy as np
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score

rng = np.random.default_rng(1)
n = 3000
core = rng.normal(size=(n, 2))
y = (core[:, 0] + core[:, 1] > 0).astype(int)

spurious = np.where(rng.random(n) < 0.95, y, 1 - y).astype(float)
X = np.column_stack([core, spurious])
m = LogisticRegression().fit(X[:2000], y[:2000])

broken = np.where(rng.random(n) < 0.5, y, 1 - y).astype(float)
reversed_ = 1 - spurious
core_only = LogisticRegression().fit(core[:2000], y[:2000])

print(f"feature agrees with label {(spurious == y).mean():.4f}")
print(f"held-out, same world      {accuracy_score(y[2000:], m.predict(X[2000:])):.4f}")
print(f"spurious link broken      {accuracy_score(y[2000:], m.predict(np.column_stack([core, broken])[2000:])):.4f}")
print(f"spurious link reversed    {accuracy_score(y[2000:], m.predict(np.column_stack([core, reversed_])[2000:])):.4f}")
print(f"model without the feature {accuracy_score(y[2000:], core_only.predict(core[2000:])):.4f}")
# feature agrees with label 0.9463
# held-out, same world      0.9900
# spurious link broken      0.9100
# spurious link reversed    0.8350
# model without the feature 0.9970

3. A model is a function. A fitted logistic regression computes a score \(s = x^{\top}w + b\) from the features, then maps it to a probability with the sigmoid \(p = 1/(1+e^{-s})\); predicting class 1 when \(p > 0.5\) is the same as predicting it when \(s > 0\). Fit one, extract its coefficients \(w\) and intercept \(b\), and reproduce a single prediction by hand from the raw numbers. Report the agreement. Here coef_[0] selects the binary model’s coefficient vector and intercept_[0] its scalar intercept. row.reshape(1, -1) makes a one-row input matrix; predict_proba returns columns in classes_ order, so column 1 corresponds to label 1 for these 0/1 labels. Use the full fitted coefficients for the calculation, not their rounded printout.

You should get: two probabilities identical to ten decimal places, from arithmetic you can do on paper.

import numpy as np
from sklearn.linear_model import LogisticRegression

rng = np.random.default_rng(1)
A = rng.normal(0, 1, size=(2000, 4))
y = (A[:, 0] + A[:, 1] > 0).astype(int)
m = LogisticRegression().fit(A[:1500], y[:1500])

w, b = m.coef_[0], m.intercept_[0]
row = A[1600]
by_hand = 1 / (1 + np.exp(-(row @ w + b)))

print(f"sklearn {m.predict_proba(row.reshape(1, -1))[0, 1]:.10f}")
print(f"by hand {by_hand:.10f}")
print(f"coefficients {np.round(w, 4)}  intercept {b:.4f}")
# sklearn 0.9911873901
# by hand 0.9911873901
# coefficients [ 7.0041  6.9494 -0.0549  0.0746]  intercept -0.0498