Face Verification as Binary Classification

Triplet loss is one way to learn face embeddings, but face verification can also be trained directly as a binary classification problem.

The model receives two face images and predicts:\[ \hat{y} = P(\text{same identity}\mid x_i,x_j) \]

The target is:\[ y= \begin{cases} 1, & \text{the images show the same person}\\ 0, & \text{the images show different people} \end{cases} \]

A Siamese network computes an embedding for each image, and a comparison head converts the two embeddings into a same-or-different probability.

Shared Face-Embedding Network

Let:\[ f_\theta(x)\in\mathbb{R}^{d} \]

be a neural network that maps a face image to a \(d\)-dimensional embedding.

For two images \(x_i\) and \(x_j\), the Siamese branches calculate:\[ e_i=f_\theta(x_i) \]\[ e_j=f_\theta(x_j) \]

Both branches share the same parameters:\[ \theta_i=\theta_j=\theta \]

This ensures that both faces are represented in the same embedding space.

An embedding dimension such as:\[ d=128 \]

is common in explanatory examples, but the method works with other dimensions.

Binary Comparison Head

The two embeddings are converted into pairwise comparison features. These features are then passed to a logistic-regression unit or a small neural network.

The final prediction is:\[ \hat{y} = \sigma(z) \]

where:\[ \sigma(z) = \frac{1}{1+e^{-z}} \]

The output can be interpreted as an estimated probability that the two images show the same identity.

Absolute-Difference Features

A simple comparison feature is the elementwise absolute difference:\[ v_k = \left| e_{i,k}-e_{j,k} \right| \]

for:\[ k=1,\ldots,d \]

The complete comparison vector is:\[ v= |e_i-e_j| \]

The logistic-regression score becomes:\[ z = \sum_{k=1}^{d}w_k \left| e_{i,k}-e_{j,k} \right| +b \]

and the prediction is:\[ \hat{y} = \sigma \left( \sum_{k=1}^{d}w_k \left| e_{i,k}-e_{j,k} \right| +b \right) \]

The learned weights determine which embedding dimensions are most useful for deciding whether two faces match.

Why Use Absolute Differences?

Absolute difference has several useful properties.

Symmetry

Swapping the two images does not change the comparison vector:\[ |e_i-e_j| = |e_j-e_i| \]

Therefore:\[ \hat{y}(x_i,x_j) = \hat{y}(x_j,x_i) \]

This is desirable because identity matching should not depend on input order.

Per-Dimension Comparison

The classifier can learn different importance weights for different embedding dimensions.

Differentiability

The absolute-value operation is differentiable almost everywhere and can be trained through backpropagation using standard subgradient handling at zero.

Weighted Distance Interpretation

The comparison score can also be understood as a learned weighted \(L_1\) distance:\[ D_W(e_i,e_j) = \sum_{k=1}^{d} a_k \left| e_{i,k}-e_{j,k} \right| \]

If the coefficients satisfy:\[ a_k\geq0 \]

then larger differences increase the learned distance.

A same-identity probability could be written as:\[ \hat{y} = \sigma \left( b-D_W(e_i,e_j) \right) \]

This makes the sign convention explicit: greater distance should generally reduce the predicted match probability.

A fully unconstrained logistic head can learn either sign convention from data, but a distance-oriented form is often easier to interpret.

Squared-Difference Features

Another symmetric comparison is:\[ v_k = (e_{i,k}-e_{j,k})^2 \]

The classifier becomes:\[ \hat{y} = \sigma \left( b+ \sum_{k=1}^{d} w_k (e_{i,k}-e_{j,k})^2 \right) \]

This resembles a learned diagonal quadratic distance.

If the weights are constrained appropriately, it can be interpreted as a weighted squared Euclidean distance.

Chi-Square-Style Comparison

For nonnegative feature components, another comparison feature is:\[ v_k = \frac{ (e_{i,k}-e_{j,k})^2 }{ e_{i,k}+e_{j,k}+\epsilon } \]

where \(\epsilon>0\) prevents division by zero.

The complete chi-square-style distance is:\[ D_{\chi^2}(e_i,e_j) = \sum_{k=1}^{d} \frac{ (e_{i,k}-e_{j,k})^2 }{ e_{i,k}+e_{j,k}+\epsilon } \]

This comparison is naturally suited to nonnegative histogram-like features. It is not automatically appropriate for arbitrary signed embeddings unless they are transformed or the formulation is modified carefully.

Combining Multiple Pairwise Features

A comparison head may use several symmetric interactions:\[ v = \begin{bmatrix} |e_i-e_j|\\ e_i\odot e_j\\ (e_i-e_j)^2 \end{bmatrix} \]

where:\[ e_i\odot e_j \]

is the elementwise product.

A neural comparison function then predicts:\[ \hat{y} = h_\phi(v) \]

where \(\phi\) represents the comparison-head parameters.

The absolute difference captures disagreement, while the elementwise product captures feature alignment.

Binary Cross-Entropy Loss

For a labeled pair:\[ (x_i,x_j,y) \]

the binary cross-entropy loss is:\[ \mathcal{L}_{\text{BCE}} = – y\log\hat{y} – (1-y)\log(1-\hat{y}) \]

Positive Pair

If:\[ y=1 \]

then:\[ \mathcal{L} = -\log\hat{y} \]

The model is encouraged to increase the predicted same-identity probability.

Negative Pair

If:\[ y=0 \]

then:\[ \mathcal{L} = -\log(1-\hat{y}) \]

The model is encouraged to reduce the predicted match probability.

For \(M\) labeled pairs:\[ J(\theta,\phi) = \frac{1}{M} \sum_{m=1}^{M} \mathcal{L}_{\text{BCE}}^{(m)} \]

Backpropagation updates both:

• The embedding-network parameters \(\theta\)
• The comparison-head parameters \(\phi\)

Constructing Positive Pairs

A positive pair contains two different images of the same identity:\[ (x_i,x_j,1) \]

where:\[ \operatorname{id}(x_i) = \operatorname{id}(x_j) \]

Useful positive pairs should contain realistic within-person variation:

• Different poses
• Different expressions
• Different lighting
• Different cameras
• Changes in appearance
• Moderate occlusion
• Different image quality

If positive pairs are nearly identical, the model may not learn the invariances needed in deployment.

Constructing Negative Pairs

A negative pair contains images of different identities:\[ (x_i,x_j,0) \]

where:\[ \operatorname{id}(x_i) \neq \operatorname{id}(x_j) \]

Random negative pairs are often easy. Once the model learns basic identity features, most random pairs can be separated with little effort.

Hard-negative mining can select different identities that the current model considers unusually similar.

Pair Sampling

The number of possible negative pairs usually far exceeds the number of positive pairs.

If identity \(k\) contains \(n_k\) images, the number of unordered positive pairs for that identity is:\[ \binom{n_k}{2} = \frac{ n_k(n_k-1) }{2} \]

The number of negative pairs across identities grows much faster.

Training therefore needs a sampling strategy, such as:

• Balanced positive and negative pairs
• Random negative sampling
• Hard-negative mining
• Semi-hard negative sampling
• Difficulty-aware curricula

The sampled training distribution need not match the real deployment distribution, but this affects probability calibration.

Class Imbalance and Calibration

A balanced training set may contain equal numbers of positive and negative pairs. A real authentication system may encounter far more impostor possibilities than genuine matches.

As a result, the raw sigmoid output may not be a well-calibrated real-world probability.

The operating threshold should be selected using representative validation data and metrics such as:

• False-accept rate
• False-reject rate
• True-accept rate
• Receiver operating characteristic
• Precision and recall
• Detection error tradeoff

A default threshold of \(0.5\) is not automatically appropriate.

Precomputing Enrollment Embeddings

Suppose a new probe image is compared with an enrolled identity.

The enrolled reference image does not change, so its embedding can be computed once:\[ e_k=f_\theta(x_k) \]

and stored.

When a new probe arrives, the system calculates only:\[ e_q=f_\theta(x_q) \]

It then evaluates:\[ \hat{y} = h_\phi(e_q,e_k) \]

This avoids repeatedly passing every enrollment image through the convolutional network.

Computational Benefit

Assume the database contains \(K\) enrolled identities.

Without precomputation, one query might require:\[ K+1 \]

embedding-network evaluations:

• One for the probe
• One for each enrolled reference

With precomputation, it requires:\[ 1 \]

new embedding evaluation, followed by \(K\) relatively inexpensive embedding comparisons.

For large databases, this provides substantial savings.

Multiple Templates per Identity

If an identity has multiple enrollment images:\[ e_{k,1},e_{k,2},\ldots,e_{k,n_k} \]

the system can:

• Compare the probe with every template
• Average normalized embeddings
• Use the minimum distance
• Average comparison scores
• Train a template-aggregation model

A normalized mean template is:\[ \bar{e}_k = \frac{ \sum_{r=1}^{n_k}e_{k,r} }{ \left\| \sum_{r=1}^{n_k}e_{k,r} \right\|_2 } \]

The best aggregation method depends on how enrollment and probe conditions vary.

Pair Classification for Verification

For one-to-one verification, a learned binary comparison head is natural:\[ (x_q,x_k) \rightarrow \hat{y} \]

The system answers whether the probe matches the claimed identity.

A decision rule is:\[ \text{accept} \quad\text{if}\quad \hat{y}\geq\tau \]

Otherwise:\[ \text{reject} \]

Because the comparison head is trained specifically for same-versus-different decisions, it may learn a more flexible boundary than a single unweighted distance threshold.

Extending the Method to Identification

For one-to-many identification, the probe can be compared with every enrolled identity:\[ \hat{y}_k = h_\phi(e_q,e_k) \]

The best candidate is:\[ k^* = \underset{k}{\operatorname{argmax}} \; \hat{y}_k \]

In an open-set system, the candidate is accepted only if:\[ \hat{y}_{k^*}\geq\tau \]

Otherwise, the result is unknown.

For large databases, a complex pair classifier may be expensive because it must be evaluated for many candidates. A common approach is:

1. Use embedding distance for fast candidate retrieval.
2. Apply a more expensive comparison model only to the top candidates.

Binary Classification Versus Triplet Loss

Both approaches can train a shared embedding network, but they optimize different objectives.

Triplet loss requires:\[ d(A,P)+\alpha\leq d(A,N) \]

Binary classification instead directly optimizes:\[ P(\text{same}\mid x_i,x_j) \]

Neither method is universally superior. Performance depends on the dataset, network, sampling strategy, loss design, and deployment requirements.

Binary Classification Versus Contrastive Loss

Contrastive loss also uses image pairs, but it usually acts directly on embedding distance:\[ \mathcal{L}_{\text{contrastive}} = yD^2 + (1-y)\max(0,m-D)^2 \]

Binary cross-entropy trains a probabilistic comparison output:\[ \mathcal{L}_{\text{BCE}} = -y\log\hat{y} -(1-y)\log(1-\hat{y}) \]

Contrastive loss explicitly shapes a metric space. Binary classification allows the comparison head to learn a potentially more flexible decision function.

Symmetry Requirements

Face comparison should normally be symmetric:\[ h(e_i,e_j)=h(e_j,e_i) \]

Absolute difference, squared difference, cosine similarity, and Euclidean distance satisfy this naturally.

Simply concatenating embeddings:

$[e_i;e_j]$

does not guarantee symmetry. A classifier could produce different outputs when the image order is reversed.

If concatenation is used, possible remedies include:

• Train on both input orders
• Average both orderings
• Use a symmetric architecture
• Prefer explicitly symmetric interaction features

Security of Stored Embeddings

Precomputed embeddings reduce computation, but they remain sensitive biometric templates.

It is not sufficient to assume that an embedding is harmless because it is not a raw photograph. Depending on the model and threat environment, embeddings may permit:

• Cross-system matching
• Membership inference
• Partial reconstruction
• Identity tracking
• Unauthorized authentication attempts

Template storage should include appropriate protections such as:

• Encryption
• Access control
• Retention limits
• Audit logging
• Separation between systems
• Revocation or replacement strategies where possible
• Legal and consent review

Deployment Pipeline

A practical verification pipeline is:

enrollment image
    ↓
face detection and alignment
    ↓
shared embedding network
    ↓
stored biometric template

enrollment image
    ↓
face detection and alignment
    ↓
shared embedding network
    ↓
stored biometric template

At verification time:

probe image
    ↓
face detection and alignment
    ↓
shared embedding network
    ↓
pairwise comparison with stored template
    ↓
calibrated threshold
    ↓
accept or reject

probe image
    ↓
face detection and alignment
    ↓
shared embedding network
    ↓
pairwise comparison with stored template
    ↓
calibrated threshold
    ↓
accept or reject

Liveness or presentation-attack detection should be treated as an additional security component rather than assumed to follow from the match score.

Common Mistakes

Using Untied Networks

Both branches must share parameters unless there is a deliberate domain-asymmetric design.

Creating Mostly Easy Negative Pairs

The model may stop learning if nearly every negative is already obvious.

Treating the Sigmoid Output as Automatically Calibrated

Balanced pair sampling changes the training prior. Validation-based calibration remains necessary.

Using an Asymmetric Comparison Unintentionally

The match result should not change when the two face images are swapped.

Recomputing Static Enrollment Features

Reference embeddings can generally be calculated once and reused.

Assuming Embeddings Are Non-Sensitive

Stored embeddings are biometric data and require strong protection.

Ignoring Detection and Alignment Errors

A comparison model cannot compensate reliably for severely incorrect face crops.

Key Takeaway

Face verification can be formulated as binary classification over image pairs:\[ (x_i,x_j) \rightarrow \hat{y} = P(\text{same identity}\mid x_i,x_j) \]

A shared Siamese network computes:\[ e_i=f_\theta(x_i), \qquad e_j=f_\theta(x_j) \]

A symmetric comparison feature such as:\[ |e_i-e_j| \]

is passed to a logistic or neural comparison head, and the complete model is trained with binary cross-entropy.

This approach provides an alternative to triplet loss. It directly optimizes the verification decision, supports one-shot enrollment, and allows fixed enrollment embeddings to be precomputed for efficient deployment.