Active Learning

Active learning labels selectively. Train a model on a small seed of labeled examples, use it to pick which unlabeled examples to send to an annotator next, retrain, repeat. The aim is to reach a given accuracy with fewer labels than labeling at random would need.

The query strategies are the interesting part: uncertainty sampling picks where the model is least confident, query by committee picks where several models disagree, expected model change picks what would move the weights most, and diversity sampling picks what is unlike anything labeled so far.

The claim attached to all of them is that they beat random sampling. That is a claim worth testing rather than assuming.

A synthetic comparison from different starting points

import numpy as np
from sklearn.datasets import make_classification
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split

X, y = make_classification(n_samples=20000, n_features=50, n_informative=30,
                           class_sep=0.5, flip_y=0.02, random_state=0)
pool_x, xte, pool_y, yte = train_test_split(X, y, train_size=15000, random_state=0)

def run(strategy, seed_size, rounds=12, batch=50):
    rng = np.random.default_rng(0)
    labeled = list(rng.choice(len(pool_y), seed_size, replace=False))
    for _ in range(rounds):
        model = LogisticRegression(max_iter=3000).fit(pool_x[labeled], pool_y[labeled])
        remaining = np.setdiff1d(np.arange(len(pool_y)), labeled)
        if strategy == "uncertainty":
            margin = np.abs(model.predict_proba(pool_x[remaining])[:, 1] - 0.5)
            pick = remaining[np.argsort(margin)[:batch]]
        else:
            pick = rng.choice(remaining, batch, replace=False)
        labeled.extend(pick.tolist())
    return LogisticRegression(max_iter=3000).fit(pool_x[labeled], pool_y[labeled]).score(xte, yte), len(labeled)

print(f"{'seed labels':>12}  {'random sampling':>16}  {'uncertainty sampling':>21}  {'total labeled':>15}")
for seed_size in (10, 30, 100, 400):
    r, n = run("random", seed_size)
    u, _ = run("uncertainty", seed_size)
    print(f"{seed_size:>12}  {r:>16.4f}  {u:>21.4f}  {n:>15}")
print(f"\n  everything labeled ({len(pool_y)} rows): "
      f"{LogisticRegression(max_iter=3000).fit(pool_x, pool_y).score(xte, yte):.4f}")
#  seed labels   random sampling   uncertainty sampling    total labeled
#           10            0.6418                 0.5714              610
#           30            0.6398                 0.6098              630
#          100            0.6400                 0.5932              700
#          400            0.6390                 0.6470             1000
#
#   everything labeled (15000 rows): 0.6622

Within each row, the two strategies use the same initial seed and total label budget: that seed plus twelve rounds of fifty labels. Across rows, the seed size and total labeled count increase. Within a row, the only difference is how those six hundred additional labels are chosen.

From a ten-label seed, uncertainty sampling reaches 0.5714 and random sampling reaches 0.6418. In this run, uncertainty sampling has accuracy lower by seven percentage points for the same number of labels. At thirty and at a hundred it is still behind. Only at a four-hundred-label seed does it win, 0.6470 against 0.6390 — and then by less than a point.

The mechanism is circular. Uncertainty sampling asks the current model where it is unsure, and a model trained on ten examples may have an unreliable boundary even where its predicted probabilities appear confident. Its sense of which points are borderline is derived from that initial estimate. It then labels points along that arbitrary boundary, learns a slightly refined version of the same wrong boundary, and asks again. Random sampling, having no opinion, covers the space.

This is an active-learning cold-start problem: the acquisition policy depends on an initial model fitted with little information. The example shows why seed size and query policy need to be compared together. Once the model is good enough to know roughly where the boundary is, asking about the boundary is useful; before that, it is asking itself.

(One dataset, one model, one seed draw. The point is not that active learning loses — but that it has to be compared against random sampling rather than assumed to beat it. Settles’s survey collects the negative results explicitly: Guo and Schuurmans found off-the-shelf strategies in batch mode “often much worse than random sampling,” and Schein and Ungar found active learning sometimes needing more labels than passive learning with the same model class — logistic regression, as here. That comparison costs one extra run.)

The labeled set is selected by the query policy

A second consequence follows from the same mechanism and survives even when active learning works.

An actively collected set reflects the query policy rather than a representative deployment sample. Cross-validation on it estimates performance on that selected distribution, and the bias relative to deployment can be upward or downward. Keep a representative evaluation set outside the acquisition loop; where evaluation uses sampled observations, account for their selection probabilities and the support the policy covers.

A changed class fraction does not by itself justify prior correction. A query policy based on features and past observations can change P(X) while leaving P(Y|X) unchanged, conditional on the acquisition history. A correctly specified conditional model does not then need a prior-odds shift. Misspecification, missing support and finite samples can still cause errors; validation on the target population and, where appropriate, propensity weighting address a different problem from label-only case-control sampling.

Making it work

• Start with a random seed big enough to be worth asking. Compare initial random seed sizes against the label budget and a held-out evaluation set; the useful size depends on the task and model.
• Mix in random picks every round. Reserving a fraction of each batch for random selection improves exploration and can preserve positive selection probabilities. It does not make an unweighted analysis representative by itself.
• Add diversity to uncertainty. Pure uncertainty sampling can pick redundant points in a batch. Adding a diversity criterion can improve coverage of the queried examples; compare the resulting policy with the random baseline.
• Hold out a random evaluation set first, before any of this starts.
• Run the random baseline alongside. It is the only way to know whether the loop is earning its complexity, and it costs one fit per round.

Where labels are expensive enough to justify the machinery, the alternatives are worth pricing too: weak supervision generates noisy labels at scale without individually annotating every example, while still requiring rule development and validation, and the question of how much complete labeling is actually needed is in full annotation.

References

• Settles, Active Learning Literature Survey, University of Wisconsin–Madison CS Technical Report 1648, 2009
• scikit-learn User Guide, Metrics and scoring