Semi-Supervised and Active Learning

When labels are expensive, two approaches make different use of a larger pool of examples. Semi-supervised learning trains with both labeled and unlabeled inputs. Active learning chooses examples whose labels will be requested from an annotator. They can be combined, but their added supervision comes from different sources: inferred targets in self-training and newly acquired labels in active learning. Unlabeled data still has collection, cleaning, and computational costs.

How unlabeled inputs can help

Unlabeled inputs describe where examples occur, but do not determine which class belongs there. Graph methods assume nearby or well-connected examples tend to share labels. Low-density separation prefers boundaries through sparse regions; a manifold assumption treats examples as lying near a lower-dimensional structure. These are related ways to connect input geometry with labels, not facts guaranteed by having more inputs. Useful gains depend on the representation, labeling task, and labeled examples.

Self-training fits a supervised model, assigns pseudo-labels to selected unlabeled examples, and refits using those inferred targets. For example, with a strict confidence threshold of 0.9, a prediction [0.04, 0.96] is accepted as class 1 while [0.3, 0.7] is not. Confidence must be meaningful for this selection; repeated training can reinforce an early error. Consistency-based methods instead encourage similar predictions under perturbations expected to preserve the label. A transformation that changes the label violates that assumption.

LabelSpreading builds a neighborhood graph, iteratively spreads class information using a normalized graph operator, and retains a contribution from the observed labels. Its alpha parameter controls this mixing: smaller values retain more of the initial label information. It is not the same predictor as ordinary k-nearest neighbors, even when both use a neighborhood of size seven. A graph component with no labeled seed can lack useful class information; finite propagation and stopping tolerances can also leave zero class mass at some points. Inspect connectivity and prediction behavior rather than assuming all unlabeled points receive reliable labels.

Compare the methods on held-out inputs

The first two methods below share a k-NN classifier; the third changes both the learning algorithm and its use of unlabeled data. This compares procedures, not an isolated causal effect of adding unlabeled inputs. The labeled-only classifier uses only the supplied labeled rows. Self-training and LabelSpreading also see the remaining training inputs, marked −1 for unknown labels. For a new point whose neighbors carry zero total class mass, the graph predictor cannot normalize a class distribution. The helper uses the labeled-only classifier for that point and reports how many rows needed this fallback. Thus the graph column evaluates LabelSpreading with a supervised fallback, not the unmodified estimator. Where mass is positive, normalization does not change the highest-scoring class. Test inputs are excluded from all fitting, so the evaluation is inductive prediction on unseen inputs. A transductive evaluation could include test inputs without their labels, but should be named and compared under that different information budget.

Each simulated seed set contains equal numbers from the two classes. This assumes such a labeled starting set is already available; it does not count the work needed to discover both classes in a real unlabeled pool. All reported accuracies average fifteen generated datasets and splits. Distances use the generated feature scales; real applications need a justified representation and preprocessing fitted within the allowed training data.

With four balanced labels, using k=4 would force every uniform-weight k-NN prediction to be a 2–2 vote. The example uses k=3 at that budget, then k=7, avoiding that artificial constant baseline. This also means the learner’s neighborhood setting changes between the first two rows. Run the Python blocks in order with NumPy and scikit-learn installed.

import numpy as np
from sklearn.datasets import make_moons, make_classification
from sklearn.semi_supervised import LabelSpreading, SelfTrainingClassifier
from sklearn.neighbors import KNeighborsClassifier
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score

from sklearn.neighbors import NearestNeighbors

def graph_predict(model, Xfit, Xnew, fallback):
    # Match the k-NN graph predictor's class-mass sum before normalization.
    neighbors = NearestNeighbors(n_neighbors=model.n_neighbors).fit(Xfit)
    index = neighbors.kneighbors(Xnew, return_distance=False)
    mass = model.label_distributions_[index].sum(axis=1)
    if not np.isfinite(mass).all():
        raise ValueError("Nonfinite graph label mass")
    unsupported = mass.sum(axis=1) == 0
    prediction = model.classes_[mass.argmax(axis=1)]
    if unsupported.any():
        prediction[unsupported] = fallback.predict(Xnew[unsupported])
    return prediction, int(unsupported.sum())

def run(make, label):
    print(f"\n=== {label} ===")
    print(f"{'labels':>7} {'kNN labelled only':>18} {'self-training':>14} {'graph+fallback':>15} {'added mean':>11} {'no-add runs':>12} {'max-iter runs':>14} {'fallback rows mean':>18}")
    for n_lab in (4, 10, 30, 100):
        sup, st, ls, added, maxed, fallback_counts = [], [], [], [], [], []
        for rep in range(15):
            X, y = make(rep)
            Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.4,
                                                  random_state=rep, stratify=y)
            g = np.random.default_rng(rep)
            idx = np.r_[g.choice(np.flatnonzero(ytr == 0), n_lab // 2, replace=False),
                        g.choice(np.flatnonzero(ytr == 1), n_lab - n_lab // 2, replace=False)]
            mask = np.full(len(ytr), -1); mask[idx] = ytr[idx]
            k = min(7, n_lab - 1)
            supervised = KNeighborsClassifier(k).fit(Xtr[idx], ytr[idx])
            sup.append(accuracy_score(yte, supervised.predict(Xte)))
            self_model = SelfTrainingClassifier(KNeighborsClassifier(k), threshold=0.9).fit(Xtr, mask)
            st.append(accuracy_score(yte, self_model.predict(Xte)))
            added.append(np.sum((mask == -1) & (self_model.transduction_ != -1)))
            maxed.append(self_model.termination_condition_ == "max_iter")
            graph = LabelSpreading(kernel="knn", n_neighbors=7, max_iter=1000).fit(Xtr, mask)
            prediction, unsupported = graph_predict(graph, Xtr, Xte, supervised)
            ls.append(accuracy_score(yte, prediction))
            fallback_counts.append(unsupported)
        print(f"{n_lab:7d} {np.mean(sup):18.4f} {np.mean(st):14.4f} {np.mean(ls):15.4f} {np.mean(added):11.1f} {np.sum(np.array(added)==0):12d} {sum(maxed):14d} {np.mean(fallback_counts):18.1f}")

run(lambda r: make_moons(n_samples=1500, noise=0.12, random_state=r),
    "two moons: clusters follow the classes")
run(lambda r: make_classification(n_samples=1500, n_features=15, n_informative=10,
                                  n_redundant=0, class_sep=0.6, flip_y=0.05, random_state=r),
    "15 dimensions, overlapping classes")
# === two moons: clusters follow the classes ===
#  labels  kNN labelled only  self-training  graph+fallback  added mean  no-add runs  max-iter runs fallback rows mean
#       4             0.7299         0.7299          0.7792         0.0           15              0              411.7
#      10             0.7414         0.7414          0.8482         0.0           15              0              239.4
#      30             0.8986         0.9753          0.9694       859.1            0              2               22.7
#     100             0.9919         0.9956          0.9946       793.7            0              0                0.4
# 
# === 15 dimensions, overlapping classes ===
#  labels  kNN labelled only  self-training  graph+fallback  added mean  no-add runs  max-iter runs fallback rows mean
#       4             0.5281         0.5281          0.5603         0.0           15              0               17.1
#      10             0.5553         0.5553          0.6229         0.0           15              0                0.0
#      30             0.6342         0.5784          0.7202       652.9            1              5                0.0
#     100             0.7409         0.7138          0.7596       447.0            0              6                0.0

On two moons with four labels, the supervised score is 0.7299 and the graph-plus-fallback score is 0.7792. The graph procedure falls back on an average of 411.7 of the 600 test rows, so this is not evidence that propagation alone classified those rows. With thirty labels, self-training reaches 0.9753 against 0.8986 for the supervised baseline. The added-label counts show whether self-training actually accepted pseudo-labels. Equal test accuracies alone cannot establish that nothing was added, since different fitted models can receive the same score. With four labels, two per class, a three-neighbor vote cannot exceed 2/3. With ten labels, five per class, an initial seven-neighbor vote cannot exceed 5/7, so it cannot cross 0.9. With more labels, unanimous local votes become possible. For these uniform-weight classifiers, predicted class probabilities are fractions of agreeing neighbors, not automatically calibrated probabilities of correctness.

The max-iteration count records runs stopped by the self-training iteration limit. The displayed results use that configured stopping rule; they are not necessarily fixed points of unlimited self-training. LabelSpreading gets a larger iteration allowance. The helper handles zero mass explicitly without dividing by zero, and warnings are not globally suppressed. Gains or losses on these datasets do not establish that all separated clusters help or all overlapping high-dimensional classes hurt. In particular, the code does not separately measure the mechanism causing each accuracy difference.

A boundary through one dense cloud

def blob(rep, n=1500):
    g = np.random.default_rng(rep)
    X = g.normal(size=(n, 2))                      # ONE dense cloud
    return X, (X[:, 0] + 0.35 * g.normal(size=n) > 0).astype(int)   # cut through the middle

print(f"{'labels':>7} {'kNN labelled only':>18} {'graph+fallback':>15} {'fallback rows mean':>18}")
for n_lab in (4, 10, 30, 100):
    sup, ls, fallback_counts = [], [], []
    for rep in range(15):
        X, y = blob(rep)
        Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.4, random_state=rep, stratify=y)
        g = np.random.default_rng(rep)
        idx = np.r_[g.choice(np.flatnonzero(ytr == 0), n_lab // 2, replace=False),
                    g.choice(np.flatnonzero(ytr == 1), n_lab - n_lab // 2, replace=False)]
        mask = np.full(len(ytr), -1); mask[idx] = ytr[idx]
        k = min(7, n_lab - 1)
        supervised = KNeighborsClassifier(k).fit(Xtr[idx], ytr[idx])
        sup.append(accuracy_score(yte, supervised.predict(Xte)))
        graph = LabelSpreading(kernel="knn", n_neighbors=7, max_iter=1000).fit(Xtr, mask)
        prediction, unsupported = graph_predict(graph, Xtr, Xte, supervised)
        ls.append(accuracy_score(yte, prediction))
        fallback_counts.append(unsupported)
    print(f"{n_lab:7d} {np.mean(sup):18.4f} {np.mean(ls):15.4f} {np.mean(fallback_counts):18.1f}")
# labels  kNN labelled only  graph+fallback fallback rows mean
#       4             0.7393          0.7731              342.7
#      10             0.7770          0.7926              140.2
#      30             0.8637          0.8472                6.4
#     100             0.8720          0.8463                0.3

In this two-dimensional generator, the probability of class 1 changes across the vertical line x₁=0, with Gaussian label noise around that boundary. It is not a deterministic division of every observed label by the line. The input density is high near the boundary, so nearby points can have different labels. This challenges a low-density-separation assumption, although local information can still be useful.

At ten labels, graph plus fallback scores 0.7926 against supervised 0.7770; at one hundred labels it scores 0.8463 against 0.8720. This is a change in the observed comparison, not a uniform failure from the smallest budget. The four label budgets show the performance of these settings over a limited range. They do not establish a permanent plateau, monotone improvement of the supervised model, or a theorem that more labels cannot help the graph method. Both methods have assumptions and can respond differently to graph size, distance scale, noise, and label placement. Geometry alone cannot confirm that an unlabeled cluster corresponds to one class; targeted labels and domain knowledge can help assess that connection.

Choose labels through an active-learning loop

Pool-based active learning repeats four operations: fit the current labeled set, score the remaining pool with an acquisition rule, obtain labels for selected examples, and refit. The simulation reads selected labels from its hidden label array as an idealized annotator. Other pool labels are unavailable to the acquisition rule. It assumes equal labeling costs and does not model annotation mistakes or delays.

Binary uncertainty sampling chooses probabilities nearest 0.5. If two candidates have probabilities 0.51 and 0.95, it queries the former. This is closeness to the model’s probability threshold, not necessarily geometric distance in the original feature units or the example with greatest eventual value. Alternatives include random sampling, disagreement among models, and batch choices that balance uncertainty with coverage or diversity.

The following comparison starts both strategies with the same ten known labels for each dataset and uses a separate test set throughout. Its ten seed datasets are paired across strategies. The balanced initialization again assumes five known examples of each class. Budget counts include those ten labels and each later queried batch; they exclude the test labels and the work of finding the initial balanced set. The test scores are recorded for analysis and do not choose the next batch.

from sklearn.linear_model import LogisticRegression

def curve(strategy, seed=0, n_rounds=14, batch=10):
    if strategy not in ("random", "uncertainty"):
        raise ValueError("Unknown acquisition strategy")
    g = np.random.default_rng(seed)
    X, y = make_classification(n_samples=4000, n_features=20, n_informative=8,
                               class_sep=0.9, flip_y=0.02, random_state=seed)
    Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.4, random_state=seed, stratify=y)
    lab = list(np.r_[g.choice(np.flatnonzero(ytr == 0), 5, replace=False),
                     g.choice(np.flatnonzero(ytr == 1), 5, replace=False)])
    accs = []
    for _ in range(n_rounds):
        m = LogisticRegression(max_iter=2000).fit(Xtr[lab], ytr[lab])
        accs.append(accuracy_score(yte, m.predict(Xte)))
        pool = np.setdiff1d(np.arange(len(ytr)), lab)
        if strategy == "random":
            pick = g.choice(pool, batch, replace=False)
        else:
            p = m.predict_proba(Xtr[pool])[:, 1]
            pick = pool[np.argsort(np.abs(p - 0.5), kind="stable")[:batch]]     # closest to the boundary
        lab.extend(pick.tolist())
    accs.append(accuracy_score(yte, LogisticRegression(max_iter=2000)
                               .fit(Xtr[lab], ytr[lab]).predict(Xte)))
    return np.array(accs)

random_runs = np.array([curve("random", seed=s) for s in range(10)])
uncertain_runs = np.array([curve("uncertainty", seed=s) for s in range(10)])
R, U = random_runs.mean(axis=0), uncertain_runs.mean(axis=0)
for i in (0, 1, 2, 4, 9, 14):
    print(f"labels {10 + 10 * i:4d}   random {R[i]:.4f}   uncertainty {U[i]:.4f}   diff {U[i] - R[i]:+.4f}")
# labels   10   random 0.6973   uncertainty 0.6973   diff +0.0000
# labels   20   random 0.7309   uncertainty 0.7262   diff -0.0047
# labels   30   random 0.7309   uncertainty 0.7622   diff +0.0313
# labels   50   random 0.7557   uncertainty 0.7822   diff +0.0264
# labels  100   random 0.7851   uncertainty 0.8068   diff +0.0217
# labels  150   random 0.8054   uncertainty 0.8178   diff +0.0124

Here uncertainty sampling reaches mean accuracy 0.8068 at one hundred labels, while random sampling reaches 0.8054 at one hundred fifty. That is a descriptive comparison of two budget points, not a demonstrated one-third saving at equal performance. Compare learning curves over labeling budgets, including a random-query baseline. An early deficit or later gain in one simulated average does not identify why an acquisition rule behaved that way. The full curve in Exercise 2 shows intermediate budgets as well as variability of paired differences. There is no universal twenty-label cutoff before uncertainty sampling becomes useful, and its advantage need not decrease steadily.

Near-duplicate queries can make a batch redundant, while uncertainty can focus on noisy, ambiguous, or atypical examples. Diversity and exploration are possible responses, not guarantees. Budget decisions also depend on annotation time, class coverage, model-refitting cost, and the value of each error type. Choose acquisition settings and stopping rules on separate validation evidence; repeatedly consulting the test curve would make it part of model selection.

Actively collected labels are generally not a representative test sample. An unweighted score on them need not estimate population performance, and its bias can have either sign, especially if those rows also trained the model. Keep a separate evaluation sample drawn from the population of interest. Selection-aware estimates require additional assumptions, recorded selection probabilities, and adequate coverage; deterministic querying may leave regions with no chance of being sampled.

Validate within the labeling budget

Validation and final evaluation consume labels too. Small validation samples can disagree with larger evaluation samples about the better procedure, especially when the performance gap is small. Record the selection rule, including ties, and evaluate the selected procedure separately. For two classifiers scored on the same examples, uncertainty in their difference depends on where their predictions disagree; the standard error of one classifier’s accuracy is not the standard error of the comparison.

The final exercise uses the same declared fallback and reserves validation inputs as well as labels from the graph fit, matching its unseen-input test setting. This avoids giving graph prediction on validation rows an extra transductive advantage. It compares the validation choice with the winner on a finite test sample, not an inaccessible population truth. More representative labels, careful audits, and related-domain evidence can improve a decision, but none makes a tiny validation set decisive. A supervised baseline is a useful comparator, not an assumption-free or automatically improving fallback.

Exercises

1. Vary the self-training confidence threshold.

import numpy as np
from sklearn.datasets import make_classification
from sklearn.semi_supervised import SelfTrainingClassifier
from sklearn.neighbors import KNeighborsClassifier
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score

def one(rep):
    X, y = make_classification(n_samples=2000, n_features=15, n_informative=8,
                               class_sep=0.7, flip_y=0.03, random_state=rep)
    Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.4, random_state=rep, stratify=y)
    g = np.random.default_rng(rep)
    idx = np.r_[g.choice(np.flatnonzero(ytr == 0), 15, replace=False),
                g.choice(np.flatnonzero(ytr == 1), 15, replace=False)]
    return Xtr, Xte, ytr, yte, idx

print(f"{'threshold':>10} {'accuracy':>9} {'pseudo-labelled':>16} {'pseudo accuracy':>16} {'empty runs':>11} {'max-iter runs':>14}")
for thr in (0.60, 0.70, 0.80, 0.90, 0.95, 0.99):
    accs, used, pacc, maxed = [], [], [], []
    for rep in range(12):
        Xtr, Xte, ytr, yte, idx = one(rep)
        mask = np.full(len(ytr), -1); mask[idx] = ytr[idx]
        st = SelfTrainingClassifier(KNeighborsClassifier(7), threshold=thr).fit(Xtr, mask)
        accs.append(accuracy_score(yte, st.predict(Xte)))
        added = (st.transduction_ != -1) & (mask == -1)
        used.append(added.sum())
        pacc.append(accuracy_score(ytr[added], st.transduction_[added]) if added.any() else np.nan)
        maxed.append(st.termination_condition_ == "max_iter")
    valid = np.array(pacc)[np.isfinite(pacc)]
    pseudo_mean = valid.mean() if len(valid) else np.nan
    print(f"{thr:10.2f} {np.mean(accs):9.4f} {np.mean(used):16.1f} {pseudo_mean:16.4f}"
          f" {len(pacc)-len(valid):11d} {sum(maxed):14d}")

base = []
for rep in range(12):
    Xtr, Xte, ytr, yte, idx = one(rep)
    base.append(accuracy_score(yte, KNeighborsClassifier(7).fit(Xtr[idx], ytr[idx]).predict(Xte)))
print(f"{'none':>10} {np.mean(base):9.4f}")
# threshold  accuracy  pseudo-labelled  pseudo accuracy  empty runs  max-iter runs
#       0.60    0.6776           1131.4           0.6788           0              0
#       0.70    0.6776           1131.4           0.6788           0              0
#       0.80    0.6623           1060.2           0.6802           0              0
#       0.90    0.5989            961.1           0.6420           0              5
#       0.95    0.5989            961.1           0.6420           0              5
#       0.99    0.5989            961.1           0.6420           0              5
#       none    0.6700

2. Inspect the complete paired active-learning curves.

print("labels random uncertainty paired_gap paired_SD")
for i in range(len(R)):
    differences = uncertain_runs[:, i] - random_runs[:, i]
    print(f"{10+10*i} {R[i]:.4f} {U[i]:.4f} {U[i]-R[i]:+.4f} {differences.std(ddof=1):.4f}")
# labels random uncertainty paired_gap paired_SD
# 10 0.6973 0.6973 +0.0000 0.0000
# 20 0.7309 0.7262 -0.0047 0.0375
# 30 0.7309 0.7622 +0.0313 0.0337
# 40 0.7429 0.7659 +0.0230 0.0336
# 50 0.7557 0.7822 +0.0264 0.0417
# 60 0.7613 0.7906 +0.0293 0.0361
# 70 0.7758 0.7933 +0.0176 0.0371
# 80 0.7737 0.7998 +0.0260 0.0301
# 90 0.7758 0.8033 +0.0276 0.0281
# 100 0.7851 0.8068 +0.0217 0.0283
# 110 0.7899 0.8088 +0.0189 0.0271
# 120 0.7959 0.8135 +0.0176 0.0238
# 130 0.7970 0.8161 +0.0191 0.0214
# 140 0.8027 0.8174 +0.0147 0.0168
# 150 0.8054 0.8178 +0.0124 0.0129

3. Reserve validation labels and evaluate the selected method.

# Uniformly request budget labels, reserving a subset for validation.
print("budget train validation supervised graph_fallback selected agreement val_ties test_ties mean_paired_SE fallback_val_mean fallback_test_mean")
for budget in (20, 40, 100, 300):
    sup_test, ls_test, selected_test, agreement, pair_se = [], [], [], [], []
    val_ties = test_ties = 0
    fallback_val, fallback_test = [], []
    for rep in range(200):
        X, y = blob(rep)
        Xpool, Xtest, ypool, ytest = train_test_split(
            X, y, test_size=0.4, random_state=rep, stratify=y)
        g = np.random.default_rng(rep)
        requested = g.choice(len(ypool), budget, replace=False)
        n_val = budget//3
        val, labeled = requested[:n_val], requested[n_val:]
        fit_rows = np.setdiff1d(np.arange(len(ypool)), val)
        mask = np.full(len(ypool), -1)
        mask[labeled] = ypool[labeled]
        sup = KNeighborsClassifier(min(7, len(labeled))).fit(Xpool[labeled], ypool[labeled])
        ls = LabelSpreading(kernel="knn", n_neighbors=7, max_iter=1000).fit(
            Xpool[fit_rows], mask[fit_rows])
        sup_correct = sup.predict(Xpool[val]) == ypool[val]
        val_prediction, nv = graph_predict(ls, Xpool[fit_rows], Xpool[val], sup)
        test_prediction, nt = graph_predict(ls, Xpool[fit_rows], Xtest, sup)
        fallback_val.append(nv); fallback_test.append(nt)
        ls_correct = val_prediction == ypool[val]
        sv, lv = sup_correct.mean(), ls_correct.mean()
        st = accuracy_score(ytest, sup.predict(Xtest))
        lt = accuracy_score(ytest, test_prediction)
        choose_ls = lv > sv  # validation ties select the supervised baseline
        sup_test.append(st); ls_test.append(lt)
        selected_test.append(lt if choose_ls else st)
        val_ties += int(lv == sv)
        test_ties += int(lt == st)
        if lt != st:
            agreement.append(choose_ls == (lt > st))
        paired = ls_correct.astype(float) - sup_correct.astype(float)
        pair_se.append(paired.std(ddof=1)/np.sqrt(n_val))
    print(budget, budget-budget//3, budget//3,
          *(round(np.mean(v), 4) for v in (sup_test, ls_test, selected_test, agreement)),
          val_ties, test_ties, round(np.mean(pair_se), 4),
          round(np.mean(fallback_val), 2), round(np.mean(fallback_test), 2))
# budget train validation supervised graph_fallback selected agreement val_ties test_ties mean_paired_SE fallback_val_mean fallback_test_mean
# 20 14 6 0.7609 0.7932 0.7826 0.5758 91 2 0.1132 0.94 96.54
# 40 27 13 0.8325 0.8193 0.8343 0.6701 62 6 0.0924 0.18 8.58
# 100 67 33 0.8659 0.842 0.8622 0.7551 42 4 0.0551 0.01 0.18
# 300 200 100 0.8763 0.8616 0.873 0.7113 34 6 0.0254 0.0 0.0

References

scikit-learn’s semi-supervised learning guide describes self-training and graph-based methods. The SelfTrainingClassifier documentation specifies confidence selection and termination diagnostics; LabelSpreading documents the graph and clamping parameters.