Face Recognition and Metric Learning

ByJu Yeon Eum

A fixed classification head predicts among the identities it was trained to recognize. Some face-recognition systems must also enroll new identities without fitting a new output class each time. An embedding model supports this by turning an image into a vector that can be stored and compared. Classification remains useful for learning that representation, as ArcFace illustrates below. The distinction is between the training objective and the recognition procedure at deployment.

Verification and identification

Verification compares a probe image with a claimed identity’s reference template. Identification searches a gallery. Closed-set identification assumes the person is enrolled; open-set identification must also reject unenrolled probes. For an unenrolled probe, testing more templates creates more opportunities for a false match. The often-used calculation \(1-(1-p)^K\) assumes \(K\) independent comparisons, all with false-match probability \(p\). Comparisons sharing the same probe are generally dependent, so a verification error rate does not determine the gallery error by itself.

How to choose an objective for a given problem is the subject of Loss Function Design: Choosing an Objective That Matches the Problem.

One-shot learning through embeddings

Learn an encoder that maps images of the same identity to nearby vectors and separates different identities under a chosen comparison rule. Enrollment with a fixed encoder then adds one or more reference vectors to a gallery. “One-shot” here means enrollment from one example of a new identity, not learning a general-purpose encoder from one photo. Multiple reference images can capture variation; a template may store several vectors or aggregate them. New enrollment need not retrain the encoder, though the index and operational evaluation must stay current.

L2 normalization is common: divide a nonzero embedding by its norm to obtain a unit vector. A zero or near-zero encoder output needs handling rather than being assumed to lie on the unit sphere. For unit vectors, \(\lVert u-v\rVert^2=2-2u^Tv\), so cosine similarity and squared Euclidean distance induce opposite but equivalent rankings. Their thresholds have different values and directions: cosine \(\geq0.75\) corresponds to squared distance \(\leq0.5\). For \(u=(1,0)\), \(v=(0.8,0.6)\), the similarity is 0.8 and squared distance is 0.4. Use the same encoder version and preprocessing for probes and stored references.

Siamese networks

A Siamese network applies shared encoder weights to two inputs, then compares their representations. The pair loss backpropagates through both uses of that encoder. Weight sharing gives a common coordinate system, but a useful identity representation still has to be learned. Independently trained encoders are not automatically compatible; separately trained models can also be explicitly aligned or trained for compatible embeddings.

Two ways to train it

The binary classification formulation. Use the absolute elementwise difference of two embeddings to predict a same-identity label of 1 versus a different-identity label of 0:

\[\hat y=\sigma\left(\sum_k w_k\left|f(x_1)_k-f(x_2)_k\right|+b\right)\]

The absolute difference makes the score symmetric in the two inputs. During training, feed the pre-sigmoid score to BCEWithLogitsLoss. Gallery embeddings can be cached here and in triplet-trained systems, leaving one encoder pass per probe. But the learned pair scorer is not automatically a distance metric or a monotone function of cosine similarity: unconstrained weights can define a different ranking. At deployment, use the trained scorer or validate a replacement comparison rule. Pair sampling also affects whether its sigmoid output is calibrated for deployment.

Triplet loss. Take an anchor \(A\), a positive \(P\) of the same identity, and a negative \(N\) of a different one. Require the anchor to be closer to the positive than to the negative by at least a margin \(\alpha\):

\[\mathcal{L}=\max\left(\lVert f(A)-f(P)\rVert^2-\lVert f(A)-f(N)\rVert^2+\alpha,\;0\right)\]

With squared distances, zero margin permits a constant embedding to achieve zero loss. A positive margin makes that collapsed solution incur loss \(\alpha\), but does not guarantee escape: if all embeddings coincide, the squared-distance gradients are still zero. This also holds for a constant unit vector, so normalization alone does not prevent collapse. A triplet strictly beyond the required margin has zero gradient. At exact equality the hinge has a kink; PyTorch’s ReLU uses a zero derivative there.

import torch
import torch.nn.functional as F

def triplet_loss(anchor, positive, negative, margin=0.2):
    """Equal nonempty (N,D) tensors; squared-distance formulation."""
    if (anchor.ndim != 2 or anchor.shape != positive.shape
            or anchor.shape != negative.shape or anchor.shape[0] == 0):
        raise ValueError("expected matching nonempty (N,D) embeddings")
    if margin < 0:
        raise ValueError("margin must be nonnegative")
    d_pos = (anchor - positive).pow(2).sum(dim=1)
    d_neg = (anchor - negative).pow(2).sum(dim=1)
    return F.relu(d_pos - d_neg + margin).mean()

a = torch.tensor([[1., 0.], [1., 0.]])
p = torch.tensor([[0.8, 0.6], [0.8, 0.6]])
n = torch.tensor([[0., 1.], [0.96, 0.28]])
print(round(triplet_loss(a[:1], p[:1], n[:1]).item(), 4))
print(round(triplet_loss(a[1:], p[1:], n[1:]).item(), 4))
print(round(triplet_loss(a, p, n).item(), 4))
# 0.0
# 0.52
# 0.26

All vectors in this example have unit length. The positive squared distance is 0.4 in both rows. Negative distances are 2.0 and 0.08, giving losses 0 and 0.52, then a batch mean of 0.26. The margin is in squared-distance units. This function differs from the default unsquared-distance formula in PyTorch’s TripletMarginLoss; exchanging the two changes the objective.

Hard negative mining

As an encoder learns to separate identities, randomly selected negatives can leave many triplets beyond the margin. Measure the active-triplet fraction rather than assuming all random triplets are uninformative. Early in training, or on difficult data, random triplets may still contribute gradients. Mining selects useful comparisons from the available examples.

Batch-hard mining chooses the farthest positive and nearest negative for each anchor within the batch. A batch with at least two identities and at least two images per identity ensures both kinds of comparison are available. The code excludes self-comparisons, averages only anchors with valid positives and negatives, and raises an error if none are valid. Singleton identities therefore contribute no anchor loss, though their embeddings can still serve as negatives for other anchors.

def batch_hard_triplet_loss(emb, labels, margin=0.2):
    """Normalized (N,D) embeddings and (N,) identity labels on the same device."""
    if emb.ndim != 2 or labels.shape != (len(emb),) or margin < 0:
        raise ValueError("expected embeddings (N,D), labels (N,), nonnegative margin")
    dist = torch.cdist(emb, emb, p=2).square()
    same = labels[:, None] == labels[None, :]
    eye = torch.eye(len(labels), dtype=torch.bool, device=emb.device)
    pos_mask = same & ~eye
    neg_mask = ~same
    valid = pos_mask.any(dim=1) & neg_mask.any(dim=1)
    if not valid.any():
        raise ValueError("no anchor has both a positive and a negative")
    hardest_pos = dist.masked_fill(~pos_mask, -torch.inf).max(dim=1).values
    hardest_neg = dist.masked_fill(~neg_mask, torch.inf).min(dim=1).values
    return F.relu(hardest_pos[valid] - hardest_neg[valid] + margin).mean()

emb = torch.tensor([[1.,0.], [0.,1.], [-1.,0.], [0.,-1.]])
labels = torch.tensor([0, 0, 1, 1])
print(round(batch_hard_triplet_loss(emb, labels).item(), 4))
# 0.2

Batch-hard is not semi-hard mining. For squared distances, a semi-hard negative satisfies \(d_{AP}^2<d_{AN}^2<d_{AP}^2+\alpha\): it is farther away than the positive but still violates the margin. A batch-hard negative can be closer than the positive. Extremely hard selections may emphasize mislabeled examples or outliers, so inspect labels and compare selection policies; restricting the search to a batch does not itself guarantee stability.

Margin-based softmax: ArcFace

ArcFace learns embeddings using a classification head over training identities. Normalize both an embedding and each class-weight vector so their dot product is \(\cos\theta_j\). Scale the logits by \(s\), and replace the true-class cosine with \(\cos(\theta_y+m)\) during training. This adds an angular margin to the cross-entropy objective:

\[\mathcal{L}=-\log\frac{e^{s\cos(\theta_y+m)}}{e^{s\cos(\theta_y+m)}+\sum_{j\ne y}e^{s\cos\theta_j}}\]

Here \(y\) is the training identity, \(m\) is measured in radians, and \(s\) controls logit scale. For \(\theta_y=30^\circ\) and \(m=0.5\), the target angle becomes about \(58.65^\circ\), reducing its cosine from about 0.866 to 0.520. The model must improve the target alignment to compensate. Practical implementations handle the angular boundary so the target transformation remains appropriate when \(\theta_y+m\) goes beyond \(\pi\). Values such as \(s=64,m=0.5\) are reported settings, not universal constants. At deployment the training-identity head can be discarded and normalized embeddings compared for previously unseen identities.

The full pipeline

A common pipeline detects a face, finds landmarks, aligns and crops it, applies the encoder’s preprocessing, extracts an embedding, and compares it with reference templates. Many checkpoints use a similarity transform based on landmarks; the appropriate alignment is part of the model’s input convention. Some models tolerate or train for different crops and poses, so alignment is not universally mandatory. Match the training and deployment pipeline, and count detection or quality-control failures in system evaluation rather than silently discarding them.

A threshold converts a score into a match decision. Higher cosine thresholds usually reduce false matches at the cost of more missed genuine matches; a distance threshold moves in the opposite direction. Choose the operating point on validation data for the intended error costs, then freeze it for testing. Authentication also requires evaluating presentation attacks separately: a low impostor-match rate does not establish resistance to photographs or replayed video.

import numpy as np

# Synthetic similarity scores, not measurements from a face model.
genuine = np.array([0.9, 0.8, 0.7])
impostor = np.array([0.85, 0.6, 0.2])
for threshold in (0.75, 0.9):
    false_match = (impostor >= threshold).mean()
    false_nonmatch = (genuine < threshold).mean()
    print(threshold, round(false_match, 4), round(false_nonmatch, 4))
# 0.75 0.3333 0.3333
# 0.9 0.0 0.6667

At 0.75, one impostor pair is accepted and one genuine pair is rejected. At 0.9, none of these three impostor pairs is accepted, but two genuine pairs are rejected. Zero errors in three comparisons is not evidence of a very low population error rate. Large galleries require suitable evaluation data and uncertainty estimates, including the dependence created by repeated subjects.

A practical fine-tuning and augmentation recipe is in Image Classification in Practice: Transfer Learning and Augmentation.

Evaluation and responsible use

For verification, report the false-match rate on impostor pairs and false-nonmatch rate on genuine pairs across thresholds, with counts and uncertainty. System-level false acceptance and rejection can additionally include capture failures and other decision stages. For open-set identification, evaluate false-positive identification per unenrolled probe and missed correct identifications for enrolled probes, specifying gallery size and rank cutoff. If the goal is generalization to new identities, keep training, validation, and test identities separate; genuine evaluation pairs still contain different images of the same evaluation identity.

Examine error rates across documented demographic groups and capture conditions when the evaluation data support those analyses, and report group sample sizes and how attributes were obtained. NIST’s demographic-effects study found differences that depend on the algorithm, dataset, and error type; high aggregate accuracy can conceal them. Gender Shades evaluated gender classification, not identity matching, so its results should not be cited as direct measurements of face-verification errors.

Before collecting or deploying biometric templates, determine the legal basis and data-handling requirements for the intended jurisdiction and use. Consent is not a universal substitute for that assessment. Define access, retention, deletion, and responses to errors before deployment. Set acceptance criteria for overall and group-specific performance according to the application’s consequences, then investigate failures against those criteria rather than treating every observed difference as automatically acceptable or automatically disqualifying.

Representation-learning objectives are covered in Self-Supervised Learning: SimCLR, BYOL, MAE, and CLIP; some use unannotated images, while CLIP uses paired text–image supervision.

References

Discover more from Insightful Data Lab

Subscribe to get the latest posts sent to your email.

Similar Posts

Questions, corrections, or additional insights?

This site uses Akismet to reduce spam. Learn how your comment data is processed.