When to Change Your Evaluation Metric or Development Set
An evaluation metric and development set define the target toward which a machine-learning team optimizes. They allow different models to be compared quickly and consistently.
Sometimes, however, a model that scores better according to the chosen metric performs worse in the application. When that happens, the target has been placed incorrectly.
If the evaluation process ranks models differently from how users and stakeholders rank them, change the evaluation metric, the evaluation data, or both.
Evaluation as Target Setting
Machine-learning development can be divided into two distinct tasks:
- Decide what success means.
- Build a system that performs well according to that definition.
The evaluation metric and development set perform the first task. They place the target.
The learning algorithm, architecture, training data, and optimizer perform the second task. They determine how effectively the model reaches the target.\[ \text{Metric and development set} \rightarrow \text{Where to aim} \]\[ \text{Training process} \rightarrow \text{How to reach it} \]
This separation is an example of orthogonalization. Defining the goal and optimizing toward it should be treated as related but distinct problems.
When Classification Error Is Not Enough
Suppose two image classifiers produce the following results:
| Model | Classification error |
|---|---|
| A | \(3\%\) |
| B | \(5\%\) |
According to ordinary classification error, model A appears better.
Now suppose model A occasionally allows explicit or otherwise unacceptable images to reach users, while model B does not. Even though model B makes more ordinary classification mistakes, it is clearly preferable from the perspective of users and the organization.
The evaluation metric says:\[ A>B \]
but actual preferences say:\[ B>A \]
This disagreement indicates that the metric does not correctly represent the application.
A metric is useful only when improving it generally produces a better real-world system.
Ordinary Classification Error
For a development set containing \(m_{\text{dev}}\) examples, ordinary classification error is:\[ \operatorname{Error} = \frac{1}{m_{\text{dev}}} \sum_{i=1}^{m_{\text{dev}}} \mathbf{1} \left( \hat{y}^{(i)}\neq y^{(i)} \right) \]
where:
- \(\hat{y}^{(i)}\) is the predicted label
- \(y^{(i)}\) is the correct label
- \(\mathbf{1}(\cdot)\) is the indicator function
The indicator function is:\[ \mathbf{1} \left( \hat{y}^{(i)}\neq y^{(i)} \right) = \begin{cases} 1, & \hat{y}^{(i)}\neq y^{(i)}\\ 0, & \hat{y}^{(i)}=y^{(i)} \end{cases} \]
This metric treats every mistake as equally costly.
That assumption may be inappropriate. In many applications, certain errors are far more serious than others.
Weighted Classification Error
A weighted metric can assign larger penalties to more harmful errors.
Define a weight \(w^{(i)}\) for each example:\[ w^{(i)} = \begin{cases} 1, & \text{ordinary example}\\ 10, & \text{high-risk example} \end{cases} \]
Then calculate:\[ \operatorname{WeightedError} = \frac{ \sum_{i=1}^{m_{\text{dev}}} w^{(i)} \mathbf{1} \left( \hat{y}^{(i)}\neq y^{(i)} \right) }{ \sum_{i=1}^{m_{\text{dev}}}w^{(i)} } \]
The denominator keeps the result normalized.
A mistake on a high-risk example now contributes ten times as much as an ordinary mistake. The weight could also be 100 or another value that better represents the relative cost.
Choosing the Weights
Weights should express the application’s preferences rather than being selected arbitrarily.
Relevant considerations include:
- User harm
- Safety consequences
- Legal or policy requirements
- Financial loss
- Frequency of the error
- Reputational damage
- Reversibility
- Manual-review cost
For example, a false negative in a medical screening system may be considered substantially more serious than a false positive. A fraud-detection system may assign different costs to missed fraud and unnecessary investigation.
The weighting scheme does not need to be perfect initially. It needs to rank candidate systems more accurately than the previous metric.
The Data Must Support the Metric
A weighted metric requires the relevant attributes to be labeled.
If high-risk examples receive larger weights, the development and test sets must identify which examples belong to that category.
This may require:
- Additional annotation
- Quality-control guidelines
- Consistent category definitions
- Review of ambiguous cases
- Updating historical evaluation data
A metric cannot incorporate an important property if that property is absent from the evaluation data.
Rank Ordering Is the Core Requirement
The primary purpose of an evaluation metric is to answer:
Given two candidate models, which one is better for this application?
An effective metric should produce a ranking that agrees with informed practical judgment.
If stakeholders consistently prefer model B while the metric prefers model A, the metric is not serving its purpose.
This can happen even when the metric is mathematically valid. The problem is not the calculation—it is that the calculation represents the wrong objective.
Metric Design and Model Training Are Separate Steps
Once the evaluation metric is changed, the training objective may also need to change. However, these decisions should be approached separately.
Step 1: Place the target
Define what should count as good performance:\[ \operatorname{WeightedError} \]
Step 2: Aim at the target
Choose a training objective that encourages the desired behavior:\[ J(\theta) = \frac{ \sum_{i=1}^{m} w^{(i)} \mathcal{L} \left( \hat{y}^{(i)},y^{(i)} \right) }{ \sum_{i=1}^{m}w^{(i)} } \]
where \(\theta\) represents the model parameters.
The evaluation metric and training objective do not have to be identical. Some real-world metrics are discontinuous or difficult to optimize directly. A differentiable surrogate loss can be used during training while the true metric is used for model selection.
First define what outcome should be preferred. Then determine how to train a model that achieves it.
Evaluation Metric Versus Training Objective
The two concepts serve different purposes.
| Component | Purpose |
|---|---|
| Evaluation metric | Compares models according to application priorities |
| Training objective | Provides a differentiable signal for updating parameters |
For example, the final metric might be a weighted classification error, while the training objective is weighted cross-entropy.
The evaluation metric answers:
Is this model better?
The training objective answers:
In which direction should the parameters move?
When the Evaluation Distribution Is Wrong
A metric can be appropriate while the development and test data are inappropriate.
Suppose two models are evaluated using high-quality images collected from the internet:
| Model | Error on curated images |
|---|---|
| A | \(3\%\) |
| B | \(5\%\) |
Model A appears superior.
After deployment, users submit images that are:
- Blurry
- Poorly framed
- Low resolution
- Taken under unusual lighting
- Partially obstructed
- Captured from unexpected angles
Model B may perform better on these real inputs even though it performed worse on the curated development set.
The problem is not necessarily the error metric. The problem is that the evaluation distribution does not represent the application.
Distribution Mismatch
Let:\[ P_{\text{dev}}(X,Y) \]
represent the development distribution and:\[ P_{\text{production}}(X,Y) \]
represent the production distribution.
If:\[ P_{\text{dev}}(X,Y) \neq P_{\text{production}}(X,Y) \]
then development performance may not predict production performance.
The development and test sets should ideally reflect the distribution on which the system needs to succeed.
The development set should contain the kinds of examples that matter in actual use, not merely the examples that are easiest to collect.
Improving the Evaluation Dataset
If production inputs are mostly informal mobile photographs, the development and test sets should include representative samples of those photographs.
Useful actions include:
- Collecting examples from actual usage
- Sampling across devices and locations
- Including common quality problems
- Preserving important rare cases
- Representing relevant user populations
- Matching expected class frequencies when appropriate
- Adding examples from anticipated future conditions
The goal is not necessarily to make every dataset identical. The goal is to make model selection predictive of real-world success.
Signs That the Metric Is Wrong
Consider changing the metric when:
- Stakeholders prefer a model with a worse score.
- Serious errors receive the same penalty as minor errors.
- The metric ignores latency, safety, fairness, or resource usage.
- Offline improvements do not improve user outcomes.
- The model exploits a weakness in the metric.
- The metric encourages undesirable behavior.
- Different error types have clearly different consequences.
Signs That the Evaluation Data Is Wrong
Consider changing the development or test set when:
- Production performance does not match offline performance.
- Important user groups are missing.
- The data comes from a different source than production data.
- Image, audio, or text quality differs substantially.
- Evaluation examples are unrealistically clean.
- Rare but critical cases are absent.
- The application has changed since the dataset was created.
- The model performs well on benchmarks but poorly in actual use.
Metric Problems and Data Problems Can Coexist
Sometimes both need to change.
For example, an evaluation system may:
- Underrepresent high-risk examples
- Treat all errors equally
- Ignore latency
- Use data from an outdated distribution
In that case, correcting only the metric or only the dataset may be insufficient.
A useful evaluation system combines:\[ \text{Representative data} + \text{Appropriate metric} \]
Do Not Wait for a Perfect Metric
It is usually better to establish a reasonable evaluation process quickly than to work for a long time without one.
Without a metric and development set:
- Model comparisons become subjective.
- Team members may optimize different objectives.
- Experiments become difficult to prioritize.
- Progress becomes hard to measure.
- Decisions take longer.
- Previous results are harder to reproduce.
An imperfect target still provides direction. If its weaknesses become apparent, it can be revised.
Establish a useful target early, but remain willing to move it when new evidence shows that it is misplaced.
Version Evaluation Changes Carefully
Changing the evaluation process affects comparisons with previous experiments.
When changing the metric or dataset, record:
- The previous definition
- The new definition
- Why the change was necessary
- Which models were reevaluated
- How historical rankings changed
- The effective date of the new standard
If possible, reevaluate important existing models under the new system. This creates a clean baseline and prevents results from incompatible evaluation versions from being compared directly.
A Practical Decision Process
When offline rankings disagree with practical preferences:
- Identify the models whose rankings conflict.
- Examine the examples responsible for the disagreement.
- Determine whether the problem is the metric, data distribution, or both.
- Define the behavior the evaluation should reward.
- Add the required labels or metadata.
- Revise the metric or dataset.
- Reevaluate existing models.
- Confirm that the new ranking matches application priorities.
- Only then modify the training process to optimize the revised target.
Key Takeaway
Your evaluation metric and development set define the target for model development. If they prefer a model that users or stakeholders consider worse, move the target: revise the metric, the evaluation distribution, or both. Establish a reasonable evaluation process early, use it to accelerate iteration, and update it whenever it stops representing real-world success.