Support Vector Machines (SVMs)

A support vector machine separates two classes with the hyperplane that leaves the widest gap. Many boundaries classify the training data correctly; the SVM picks the one whose nearest points on either side are as far away as possible.

\[ f(x) = w \cdot x + b, \qquad \hat{y} = \text{sign}(f(x)), \qquad \text{margin} = \frac{2}{\|w\|} \]

In the hard-margin case the points that sit on the margin are the support vectors, and they alone determine the boundary — move a distant point and nothing changes. A soft-margin SVM allows some points inside the margin or on the wrong side, with \(C\) controlling how much that is penalized; those points are support vectors too, along with the ones exactly on the margin, and the points beyond the margin on the correct side still have no influence on the boundary. The kernel trick handles boundaries that no hyperplane can draw by working in a higher-dimensional space without constructing it.

Two things commonly said about SVMs deserve checking.

The output is a distance, not a probability

import warnings
import numpy as np
from sklearn.datasets import make_classification
from sklearn.svm import SVC

warnings.filterwarnings("ignore")
X, y = make_classification(n_samples=3000, n_features=10, n_informative=6,
                           class_sep=0.3, flip_y=0.05, random_state=0)

plain = SVC(kernel="rbf", random_state=0).fit(X, y)
print(f"  support vectors {plain.n_support_.sum()} of {len(y)} training rows"
      f" - the model keeps {plain.n_support_.sum() / len(y):.1%} of them")

scaled = SVC(kernel="rbf", probability=True, random_state=0).fit(X, y)
"""probability=True fits Platt scaling by internal 5-fold cross-validation, so the
   cross-validation estimates calibration parameters; prediction applies the fitted mapping to the final SVM scores."""
by_margin = scaled.predict(X)
by_probability = (scaled.predict_proba(X)[:, 1] >= 0.5).astype(int)
disagree = by_margin != by_probability
print(f"  predict() and predict_proba() >= 0.5 disagree on {disagree.mean():.4f}"
      f" of the training rows ({disagree.sum()} of {len(y)})")
print(f"  on those rows the margin runs from {scaled.decision_function(X)[disagree].min():+.4f}"
      f" to {scaled.decision_function(X)[disagree].max():+.4f}")

print("\n  the model size grows with the data, and so does the kernel matrix")
print(f"{'rows':>7}  {'support vectors':>16}  {'share kept':>11}  {'kernel matrix entries':>22}")
for n in (500, 1000, 2000, 4000, 8000):
    Xn, yn = make_classification(n_samples=n, n_features=10, n_informative=6,
                                 class_sep=0.3, flip_y=0.05, random_state=0)
    sv = SVC(kernel="rbf", random_state=0).fit(Xn, yn).n_support_.sum()
    print(f"{n:>7}  {sv:>16}  {sv / n:>11.1%}  {n * n:>22,}")
#   support vectors 1445 of 3000 training rows - the model keeps 48.2% of them
#   predict() and predict_proba() >= 0.5 disagree on 0.0083 of the training rows (25 of 3000)
#   on those rows the margin runs from +0.0052 to +0.0424
#
#   the model size grows with the data, and so does the kernel matrix
#    rows   support vectors   share kept   kernel matrix entries
#     500               349        69.8%                 250,000
#    1000               573        57.3%               1,000,000
#    2000              1204        60.2%               4,000,000
#    4000              2358        59.0%              16,000,000
#    8000              3815        47.7%              64,000,000

decision_function returns how far a point sits from the boundary, in units of the margin. That is a ranking, not a risk: a value of 2.0 does not mean anything in probability terms, and two SVMs fitted with different \(C\) produce values on different scales.

scikit-learn’s SVC estimates the Platt calibration parameters using internal cross-validation. At prediction time it applies that mapping to the final fitted SVM’s decision values. The mapped probability can cross 0.5 at a different score from the SVM’s class boundary, so predict and thresholded predict_proba can disagree; this is not inference by a different set of subset-fitted SVMs.

The run measures it. Twenty-five of three thousand training rows are classified one way by predict and the other way by thresholding predict_proba at 0.5. Their margins run from +0.0052 to +0.0424 — all just on the positive side of the boundary, which is where the original class boundary and the calibrated 0.5 boundary differ in this run.

In scikit-learn 1.9 the probability parameter is deprecated and will be removed in 1.11; the recommended replacement is CalibratedClassifierCV(SVC(), ensemble=False), which makes the two-stage structure explicit instead of hiding it inside the estimator. Either way, if the output needs to be a probability, calibration is a separate step you are performing, and it should be fitted and evaluated as one.

“Only support vectors” is not a memory guarantee

It is true that the fitted model stores only the support vectors. The claim that this makes SVMs memory-efficient depends on how many there are.

On this data the model keeps 48.2% of the training rows, and the table shows the share staying between 47% and 70% as the sample grows. The model is roughly linear in the data size, not constant — on noisy or overlapping problems most points end up inside or near the margin, and nearly everything is a support vector.

Training is the harder constraint. Kernel SVMs work with pairwise similarities, so the kernel matrix has \(n^2\) entries — sixty-four million at eight thousand rows — and scikit-learn documents the training complexity as between quadratic and cubic in the number of samples. That is the reason SVMs faded from large-scale tabular work while remaining excellent on small, high-dimensional problems where \(n\) is in the thousands and features are in the tens of thousands.

Linear SVMs escape this entirely, since the boundary can be stored as a single weight vector and solvers exist that never form the kernel matrix. If a linear boundary suffices, use LinearSVC and the scaling problem disappears.

Using one

• Scale the features. Kernels are built from distances and inner products, so an unscaled column in thousands dominates everything — the effect normalization measures at nine accuracy points for distance-based models.
• Tune \(C\) and the kernel width together. They trade against each other, so a grid over both is the minimum; tuning one at a time finds a worse optimum.
• Inspect the support-vector count in context. It affects inference cost and reflects C, the margin and the data geometry. A large fraction is not proof of noise or excessive C: a very small C can place even well-separated observations inside the margin, making all of them support vectors. Diagnose with validation and parameter comparisons.
• Try a linear model first. Where a linear model with the right transformed features does the job, it does it faster and tells you what it learned.

And against the tree families, the division is mostly about data shape: trees and their ensembles handle mixed types, missing values and large \(n\) comfortably, while kernel SVMs remain strong where the sample is small and the feature space is wide.

References

• scikit-learn User Guide, Support Vector Machines
• scikit-learn API reference, SVC
• scikit-learn User Guide, Probability calibration