Using a Single-Number Evaluation Metric

Machine-learning development is an empirical and iterative process:\[ \text{Idea} \rightarrow \text{Implementation} \rightarrow \text{Experiment} \rightarrow \text{Evaluation} \rightarrow \text{New idea} \]

To move through this cycle efficiently, a team needs a clear way to decide whether a new model is better than the previous one.

A single-number evaluation metric compresses the most important aspects of performance into one value, making model comparison and selection faster.

A development set tells you where to evaluate, while a single-number metric tells you which model is better.

Why One Number Helps

Suppose a team trains ten candidate models with different:

• Architectures
• Hyperparameters
• Optimizers
• Training datasets
• Regularization methods
• Decision thresholds

If each model produces several metrics, selecting one can become slow and subjective.

A single primary metric provides a consistent ranking:\[ M_1>M_2 \quad\Rightarrow\quad \text{Model 1 is preferred} \]

or, for an error metric:\[ E_1<E_2 \quad\Rightarrow\quad \text{Model 1 is preferred} \]

This makes it easier to:

• Compare experiments
• Select a baseline
• Prioritize promising ideas
• Automate model selection
• Communicate progress
• Avoid subjective arguments

Precision

Precision measures the reliability of positive predictions.

For a binary classifier:\[ \operatorname{Precision} = \frac{TP}{TP+FP} \]

where:

• \(TP\) is the number of true positives
• \(FP\) is the number of false positives

For an image classifier, precision answers:

Of the images predicted to contain a cat, what fraction actually contain a cat?

If precision is:\[ 95\% \]

then approximately 95% of the model’s positive predictions are correct.

High precision means the classifier rarely labels a negative example as positive.

Recall

Recall measures how successfully the model finds actual positive examples:\[ \operatorname{Recall} = \frac{TP}{TP+FN} \]

where \(FN\) is the number of false negatives.

Recall answers:

Of all the images that actually contain a cat, what fraction did the model identify?

If recall is:\[ 90\% \]

then the classifier detects approximately 90% of the real cat images.

High recall means the model misses relatively few positive examples.

The Precision–Recall Tradeoff

Precision and recall often move in opposite directions.

If a classifier uses a high threshold before predicting “cat”:

• It makes fewer positive predictions.
• Those predictions may be more reliable.
• Precision may rise.
• Recall may fall.

If the threshold is lowered:

• The model identifies more cat images.
• It may also accept more non-cat images.
• Recall may rise.
• Precision may fall.

Consider two classifiers:

Classifier	Precision	Recall
A	\(95\%\)	\(90\%\)
B	\(98\%\)	\(85\%\)

Classifier A has better recall, while classifier B has better precision. With two separate metrics, there is no automatic winner.

Why the Arithmetic Mean Is Not Ideal

One possible combination is the arithmetic mean:\[ \frac{P+R}{2} \]

However, this can produce a seemingly reasonable score even when one component is very poor.

Suppose:\[ P=1.0 \]

and:\[ R=0.1 \]

The arithmetic mean is:\[ \frac{1.0+0.1}{2}=0.55 \]

A score of 0.55 may conceal the fact that recall is only 10%.

A useful combined metric should require both precision and recall to be reasonably strong.

F1 Score

The F1 score combines precision and recall using their harmonic mean:\[ F_1 = \frac{2} {\frac{1}{P}+\frac{1}{R}} \]

This is equivalent to:\[ F_1 = 2\frac{PR}{P+R} \]

The harmonic mean is strongly influenced by the smaller value. A model receives a high F1 score only when both precision and recall are high.

F1 Example

For classifier A:\[ P_A=0.95 \]\[ R_A=0.90 \]

Therefore:\[ F_{1,A} = 2 \frac{(0.95)(0.90)} {0.95+0.90} \approx0.924 \]

For classifier B:\[ P_B=0.98 \]\[ R_B=0.85 \]

Therefore:\[ F_{1,B} = 2 \frac{(0.98)(0.85)} {0.98+0.85} \approx0.910 \]

The F1 scores are:

Classifier	Precision	Recall	F1
A	\(0.95\)	\(0.90\)	\(0.924\)
B	\(0.98\)	\(0.85\)	\(0.910\)

If F1 appropriately represents the application’s priorities, classifier A is preferred.

F1 Requires Both Metrics to Be Strong

Consider three models:

Model	Precision	Recall	F1
A	\(0.90\)	\(0.90\)	\(0.90\)
B	\(1.00\)	\(0.50\)	\(0.67\)
C	\(0.50\)	\(1.00\)	\(0.67\)

Models B and C are excellent on one dimension but weak on the other. F1 penalizes this imbalance.

F1 is useful when precision and recall are both important and neither should become extremely poor.

F1 Is Not Always the Right Metric

F1 treats precision and recall symmetrically. Some applications value one more than the other.

For example:

• Medical screening may prioritize recall.
• Spam filtering may prioritize precision.
• Fraud detection may place different costs on each error type.
• Search systems may care about ranked relevance.

A generalized metric called \(F_\beta\) can weight recall differently:\[ F_\beta = (1+\beta^2) \frac{PR} {\beta^2P+R} \]

When:\[ \beta>1 \]

recall receives more weight.

When:\[ \beta<1 \]

precision receives more weight.

The metric should reflect the application rather than being chosen merely because it is common.

Combining Performance Across Regions

Suppose a classifier operates in four geographic markets:

• United States
• China
• India
• Other regions

A team may track one error rate per region:

Model	US	China	India	Other
A	\(3.0\%\)	\(6.0\%\)	\(5.0\%\)	\(4.0\%\)
B	\(4.0\%\)	\(4.5\%\)	\(5.5\%\)	\(4.5\%\)
C	\(3.5\%\)	\(4.0\%\)	\(4.5\%\)	\(4.0\%\)

The regional values remain useful for diagnosis, but comparing many models using four numbers each is cumbersome.

A simple average produces one metric:\[ E_{\text{average}} = \frac{1}{4} \sum_{r=1}^{4}E_r \]

For model A:\[ E_A = \frac{3.0+6.0+5.0+4.0}{4} = 4.5\% \]

For model B:\[ E_B = \frac{4.0+4.5+5.5+4.5}{4} = 4.625\% \]

For model C:\[ E_C = \frac{3.5+4.0+4.5+4.0}{4} = 4.0\% \]

According to average regional error, model C is best.

Weighted Geographic Average

An unweighted average treats all regions equally. If production traffic differs substantially, a weighted average may be more appropriate.

Let \(q_r\) represent the expected traffic share from region \(r\):\[ \sum_rq_r=1 \]

Then:\[ E_{\text{weighted}} = \sum_rq_rE_r \]

For example:

Region	Traffic weight
US	\(0.40\)
China	\(0.30\)
India	\(0.20\)
Other	\(0.10\)

The combined metric becomes:\[ E_{\text{weighted}} = 0.40E_{\text{US}} + 0.30E_{\text{China}} + 0.20E_{\text{India}} + 0.10E_{\text{Other}} \]

This better reflects the average experience across actual usage.

Alternatively, equal weighting may be appropriate when each region is strategically important regardless of traffic volume.

Average Performance Can Hide Failures

A single average may conceal unacceptable behavior in one subgroup.

Suppose a model has excellent average performance but performs very poorly in one important region. Selecting it solely by average error may be inappropriate.

A useful solution is:

• Use average error as the optimizing metric.
• Add minimum subgroup-performance requirements as satisficing metrics.

For example:\[ \min E_{\text{average}} \]

subject to:\[ E_r\leq8\% \quad \text{for every region }r \]

This preserves a clear primary ranking while preventing severe subgroup failures.

Keep Diagnostic Metrics

Using one primary metric does not mean discarding all other measurements.

A healthy evaluation system can have:

• One metric for model selection
• A dashboard of diagnostic metrics
• Safety and operational constraints
• Per-class and per-group breakdowns

For example:

Primary metric

\[ F_1 \]

Diagnostic metrics

• Precision
• Recall
• False-positive rate
• False-negative rate
• Per-region error
• Per-device error
• Calibration
• Latency

Use one number to decide and multiple numbers to understand.

This distinction prevents the model-selection process from becoming ambiguous while preserving the information needed for debugging and risk management.

The Metric Must Be Calculated on a Fixed Development Set

A model score is meaningful only in relation to the data on which it was measured.

The complete model-selection target consists of:\[ \text{Development set} + \text{Evaluation metric} \]

If either changes between experiments, results may not be comparable.

For reliable iteration:

• Use the same development examples.
• Use the same preprocessing.
• Use the same metric implementation.
• Use the same decision threshold unless threshold tuning is part of the comparison.
• Document intentional evaluation changes.

Single-Number Metrics Speed Up Teams

Without a primary metric, different team members may prefer different models:

• One prefers higher recall.
• Another prefers lower latency.
• Another focuses on one geographic region.
• Another prefers a smaller model.

A predefined metric makes the decision rule explicit before the results are known.

This improves:

• Experiment throughput
• Reproducibility
• Communication
• Automation
• Accountability
• Prioritization

It also reduces the temptation to select whichever metric makes a preferred model look best.

When a Single Number Is Difficult

Some objectives should not be combined using an arbitrary weighted sum.

For example, a model may need:

• Maximum accuracy
• Latency below 100 ms
• Memory below 1 GB
• No unacceptable safety violations

In such cases, one metric can be optimized while the others become threshold requirements:\[ \max\operatorname{Accuracy} \]

subject to:\[ \operatorname{Latency}\leq100\text{ ms} \]\[ \operatorname{Memory}\leq1\text{ GB} \]

This still creates a clear selection rule without forcing fundamentally different concerns into one artificial formula.

Designing a Primary Evaluation Metric

A practical process is:

1. List the outcomes that matter.
2. Identify which outcomes should be optimized continuously.
3. Identify which only need to satisfy thresholds.
4. Decide how classes, regions, or users should be weighted.
5. Define the metric mathematically.
6. Test it on several representative models.
7. Verify that its rankings match practical preferences.
8. Keep secondary metrics for diagnosis.
9. Revise the metric if it stops reflecting real-world goals.

Common Mistakes

Tracking many metrics without a decision rule

This makes model selection slow and subjective.

Selecting the metric after seeing the results

This encourages choosing whichever metric favors a preferred model.

Averaging incompatible values

Accuracy, latency, and memory cannot be meaningfully averaged without justified scaling and weights.

Ignoring subgroup failures

A strong average can hide unacceptable performance for an important group.

Treating F1 as universally correct

F1 assumes precision and recall should be balanced symmetrically.

Changing the development set between experiments

Scores become difficult to compare when the underlying evaluation data changes.

Key Takeaway

A fixed development set and a single primary evaluation metric make it possible to compare models quickly and consistently. Metrics such as F1 can combine related objectives like precision and recall, while averages or weighted averages can combine performance across groups. Retain secondary metrics for diagnosis, but define one clear rule for deciding which model is better.