Full Annotation

Full annotation means every sample within the defined scope carries every label the task requires. For object detection that is a box and a class for each relevant object; for classification it is a label on each image. Partial annotation leaves some of them out.

The word “full” is task-relative. An image can be fully annotated for classification and not for detection, and the annotation guidelines are what define the scope. Completeness is also not quality — a fully annotated dataset can be completely wrong, which is label noise rather than partial annotation.

The usual reason given for insisting on completeness is that unlabeled positives get treated as negatives. That is right, and what it does to a model is worth seeing.

Missing positive labels change the learning target

import numpy as np
from sklearn.datasets import make_classification
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import precision_score, recall_score, roc_auc_score
from sklearn.model_selection import train_test_split

X, y = make_classification(n_samples=20000, n_features=12, n_informative=6,
                           weights=[0.85], flip_y=0.02, random_state=0)
xtr, xte, ytr, yte = train_test_split(X, y, test_size=0.4, random_state=0)
rng = np.random.default_rng(0)
"""Partial annotation: some true positives were never looked at, so they sit in
   the training data labeled 0. The test set keeps its true labels."""

print(f"{'positives left unlabeled':>26}  {'AUROC':>7} {'precision@0.5':>14} {'recall@0.5':>11}"
      f" {'mean predicted risk':>20}")
for missed in (0.0, 0.2, 0.4, 0.6, 0.8):
    labels = ytr.copy()
    positives = np.flatnonzero(ytr == 1)
    labels[rng.choice(positives, int(len(positives) * missed), replace=False)] = 0
    p = LogisticRegression(max_iter=3000).fit(xtr, labels).predict_proba(xte)[:, 1]
    print(f"{missed:>26.0%}  {roc_auc_score(yte, p):>7.4f} {precision_score(yte, p >= 0.5):>14.4f}"
          f" {recall_score(yte, p >= 0.5):>11.4f} {p.mean():>20.4f}")
print(f"\n  the true positive rate in the test set is {yte.mean():.4f}")
#   positives left unlabeled    AUROC  precision@0.5  recall@0.5  mean predicted risk
#                         0%   0.9309         0.7813      0.6341               0.1624
#                        20%   0.9312         0.8326      0.4629               0.1299
#                        40%   0.9320         0.8830      0.2596               0.0977
#                        60%   0.9320         0.8962      0.0743               0.0651
#                        80%   0.9307         0.8000      0.0031               0.0328
#
#   the true positive rate in the test set is 0.1599

Read the columns in order of how surprising they are.

Recall collapses, from 0.6341 to 0.0031. At 80% of positives unlabeled the model has essentially stopped predicting the positive class. This is the expected damage and it is severe.

Precision rises from 0.7813 to a peak of 0.8962, then falls to 0.8000 at 80% missing positives. The model predicts very few positives at that last setting. Precision alone would miss the severe recall loss, and the trajectory is not monotonically improving.

AUROC stays close to 0.93 in this experiment: 0.9309, 0.9312, 0.9320, 0.9320 and 0.9307. This fitted model preserves the ordering approximately under the simulated random omissions. It does not establish that every learner or missing-label process preserves ranking.

Mean predicted risk falls from 0.1624 to 0.0328, tracking the recorded positive rate rather than the true 0.1599. The target has changed: the model learns whether a positive was recorded. If a true positive is recorded with constant probability c and negatives are never recorded as positive, then the recorded-positive probability is q(x) = c p(x). Missing positives contaminate the recorded-negative class; they do not merely change the prior.

Repair depends on the missing-label process

Under that constant-probability assumption, an accurate estimate of q(x) can be converted to p(x) = q(x)/c when c is known or identifiable from additional information. This is the positive-and-unlabeled setting described by Elkan and Noto. It is different from Bayesian correction for case-control sampling: an intercept or prior-odds adjustment does not generally repair missing positive labels. For example, with a true prior of 0.2 and half of positives recorded, a true risk of 0.8 becomes 0.4; prior-odds correction returns 0.6, whereas division by 0.5 recovers 0.8.

The intended use still matters. A ranking system may tolerate probability distortion if its ranking is validated against trustworthy labels. A system that consumes probabilities needs evidence that the missing-label mechanism and the probability model support the proposed correction. The experiment above gives a useful diagnostic pattern, not permission to ignore completeness.

Estimating the recording probability is itself an annotation or identification problem. It must not be confused with the fraction of recorded negatives that turn out to be positive. If difficult positives are missed more often, c depends on the input and division by one constant is insufficient. Even under random omission, finite-sample or model errors in q remain after correction.

The evaluation set is the part that must be complete

The experiment above changes the training labels and evaluates against complete test labels. Corrupting the evaluation labels is a different operation. For fixed predictions, randomly retaining each true positive with probability c reduces population precision by that factor, leaves population recall unchanged, and generally changes AUROC because the recorded negatives now include hidden positives. In a 100-row example with 40 positives and a perfect score, hiding 20 positives gives precision 0.5, recall 1.0 and AUROC 0.875. These directions differ from the training experiment.

Incomplete evaluation labels therefore require their own audit and missingness assumptions. Sampling noise and selective omissions can change the observed pattern further. A clean evaluation set is valuable both for measuring the fitted model and for assessing any correction to its training target.

When completeness is worth its cost

• Dense prediction tasks. With a loss that treats unannotated regions as background, a missing object can become an incorrect background training signal. Partial annotation requires a training objective and annotation scope that account for the missing labels.
• Benchmarks and evaluation sets, for the reason above.
• High-stakes decisions, where the probability itself drives the action and a shifted base rate changes who gets treated or investigated.
• Error analysis. You cannot count false negatives against labels that were never assigned.

Everywhere else the alternatives are real: weak supervision generates labels at scale from rules, active learning spends the annotation budget selectively, and semi-supervised methods use the unlabeled rows directly. The honest framing is that full annotation is one purchase among several, priced against what the model’s output has to mean.

References

• King & Zeng, Logistic Regression in Rare Events Data, Political Analysis 9(2), 2001
• scikit-learn User Guide, Metrics and scoring
• Elkan and Noto (2008), Learning Classifiers from Only Positive and Unlabeled Data