Optimizing and Satisficing Metrics in Machine Learning
A machine-learning system often needs to satisfy several objectives simultaneously. A model may need to be accurate, fast, memory-efficient, reliable, fair, and inexpensive to operate.
Combining every concern into a single weighted formula can be difficult and artificial. A useful alternative is to divide the evaluation criteria into:
- One optimizing metric
- One or more satisficing metrics
Optimize the quality that should be as good as possible, while requiring every other important quality to meet an acceptable threshold.
The Problem with Multiple Metrics
Suppose three image classifiers are evaluated using accuracy and inference time:
| Classifier | Accuracy | Inference time |
|---|---|---|
| A | \(94\%\) | 80 ms |
| B | \(95\%\) | 95 ms |
| C | \(97\%\) | 1,500 ms |
Which model is best?
- Classifier C has the highest accuracy.
- Classifier A is the fastest.
- Classifier B offers an intermediate balance.
There is no obvious answer until the application’s priorities are stated clearly.
Combining Metrics with a Weighted Sum
One approach is to define a single score:\[ \operatorname{Score} = \operatorname{Accuracy} – \lambda\cdot\operatorname{Runtime} \]
where \(\lambda\) controls the tradeoff between accuracy and speed.
This approach can work, but selecting \(\lambda\) may be difficult.
Questions immediately arise:
- How much accuracy is one millisecond worth?
- Should the relationship be linear?
- Is reducing latency from 1,500 ms to 500 ms equivalent to reducing it from 100 ms to 99 ms?
- Do users perceive meaningful differences below a certain threshold?
- Will the correct weight change across hardware platforms?
A weighted sum may create a mathematical tradeoff that does not reflect actual user preferences.
Optimizing Metric
An optimizing metric is the quantity that should be made as good as possible.
Examples include:
- Accuracy
- F1 score
- Recall
- Revenue
- User engagement
- Ranking quality
- Average precision
If accuracy is the optimizing metric, the goal is:\[ \max_{\theta} \operatorname{Accuracy}(\theta) \]
where \(\theta\) represents a candidate model or its parameters.
There is no point at which additional improvement is considered irrelevant. Higher accuracy remains preferable, assuming all constraints are satisfied.
Satisficing Metric
A satisficing metric only needs to meet an acceptable threshold.
Examples include:
- Inference time below 100 ms
- Memory consumption below 1 GB
- Model size below 50 MB
- False-positive rate below \(0.1\%\)
- Cost below a specified amount
- Availability above \(99.9\%\)
Once the threshold is satisfied, additional improvement is less important for model selection.
For inference time:\[ \operatorname{Runtime}(\theta) \leq100\text{ ms} \]
A model running in 50 ms satisfies the requirement, but it is not necessarily preferred over one running in 95 ms if the latter is more accurate.
A satisficing metric defines “good enough,” while an optimizing metric determines the winner among the acceptable candidates.
Constrained Model Selection
The image-classification problem can be written as:\[ \max_{\theta} \operatorname{Accuracy}(\theta) \]
subject to:\[ \operatorname{Runtime}(\theta) \leq100\text{ ms} \]
First, eliminate models that violate the runtime constraint:
| Classifier | Accuracy | Runtime | Eligible? |
|---|---|---|---|
| A | \(94\%\) | 80 ms | Yes |
| B | \(95\%\) | 95 ms | Yes |
| C | \(97\%\) | 1,500 ms | No |
Classifier C has the highest accuracy but is infeasible. Between the remaining candidates, classifier B has the best optimizing metric.
Therefore:\[ \text{Selected model}=B \]
Why Thresholds Often Match Real Requirements
Suppose users do not notice a meaningful difference between 50 ms and 95 ms, but they find delays above 100 ms disruptive.
In that situation, latency naturally behaves as a constraint:\[ \text{Latency}\leq100\text{ ms} \]
Trying to optimize it indefinitely would waste resources or sacrifice accuracy for improvements users do not value.
This pattern appears frequently:
- Below a latency threshold, the application feels responsive.
- Below a memory threshold, the model fits on the device.
- Below a cost threshold, deployment is economically viable.
- Above an availability threshold, reliability is acceptable.
The exact value still requires careful selection, but the threshold structure is often more interpretable than a weighted sum.
General Formulation
Suppose there are \(N\) important metrics:\[ M_1,M_2,\ldots,M_N \]
Choose one metric as the optimizing objective:\[ \max_{\theta}M_1(\theta) \]
Then treat the others as constraints:\[ M_2(\theta)\geq\tau_2 \]\[ M_3(\theta)\leq\tau_3 \]\[ \vdots \]\[ M_N(\theta)\geq\tau_N \]
The inequality direction depends on the metric:
- Accuracy should be above a threshold.
- Latency should be below a threshold.
- Memory should be below a threshold.
- Reliability should be above a threshold.
- Error rate should be below a threshold.
A candidate is eligible only if it satisfies every constraint.
Feasible Set
The constraints define the feasible set:\[ \mathcal{F} = \left\{ \theta: M_k(\theta) \text{ satisfies its threshold for all }k\geq2 \right\} \]
Model selection then becomes:\[ \theta^* = \operatorname*{arg\,max}_{\theta\in\mathcal{F}} M_1(\theta) \]
This produces a clear decision procedure:
- Reject models that violate any satisficing threshold.
- Compare the remaining models using the optimizing metric.
- Select the model with the best optimizing score.
Wake-Word Detection Example
A wake-word system listens for phrases that activate a voice assistant.
It must perform two distinct tasks well:
- Activate when the wake phrase is spoken.
- Avoid activating when it is not spoken.
Possible metrics include:
- Wake-word detection rate
- False activations per 24 hours
The optimizing objective might be:\[ \max_{\theta} \operatorname{DetectionRate}(\theta) \]
subject to:\[ \operatorname{FalseActivations}_{24h}(\theta) \leq1 \]
The system should detect as many genuine wake phrases as possible, but it must not activate incorrectly more than once per day on average.
Among models satisfying this constraint, choose the one with the highest detection rate.
Why False Activations Fit a Satisficing Metric
A false-activation rate of once every hour is unacceptable. Once every day may be acceptable. Once every month is better, but users may value additional detection accuracy more than further reductions beyond the acceptable threshold.
The model-selection priority can therefore be expressed as:
- Meet the false-activation requirement.
- Maximize genuine detection.
This is easier to interpret than assigning an arbitrary numerical exchange rate between a missed wake word and one false activation.
Multiple Satisficing Metrics
A real system may have several constraints.
For an on-device classifier:\[ \max_{\theta} \operatorname{Accuracy}(\theta) \]
subject to:\[ \operatorname{Latency}(\theta) \leq100\text{ ms} \]\[ \operatorname{ModelSize}(\theta) \leq50\text{ MB} \]\[ \operatorname{Memory}(\theta) \leq500\text{ MB} \]\[ \operatorname{EnergyUse}(\theta) \leq E_{\max} \]
A model is considered only if it meets all four requirements. Accuracy then selects the best eligible model.
Example with Several Constraints
| Model | Accuracy | Latency | Size | Eligible? |
|---|---|---|---|---|
| A | \(94.0\%\) | 70 ms | 40 MB | Yes |
| B | \(95.2\%\) | 92 ms | 45 MB | Yes |
| C | \(96.1\%\) | 140 ms | 48 MB | No |
| D | \(95.8\%\) | 88 ms | 75 MB | No |
Assume the requirements are:\[ \operatorname{Latency}\leq100\text{ ms} \]
and:\[ \operatorname{Size}\leq50\text{ MB} \]
Models C and D are rejected. Between A and B, model B has greater accuracy and is selected.
Choosing the Optimizing Metric
The optimizing metric should represent the dimension on which continued improvement remains valuable after all constraints are satisfied.
Possible choices include:
- Accuracy when every correct prediction matters
- Recall when missing positive cases is the main concern
- Precision when false positives are particularly costly
- Revenue when business value is the primary outcome
- NDCG or another ranking metric for search systems
- User retention for recommendation systems
Selecting an optimizing metric is an application decision, not merely a technical one.
Choosing Satisficing Thresholds
Thresholds should be based on operational or user requirements.
Sources include:
- User research
- Service-level objectives
- Hardware limitations
- Safety standards
- Legal requirements
- Product constraints
- Infrastructure budgets
- Historical baselines
A threshold should be:
- Measurable
- Relevant
- Realistically achievable
- Stable enough to guide development
- Strict enough to reject unusable models
An arbitrary threshold may lead the team toward a technically valid but practically unsuitable system.
Optimizing and Satisficing Metrics Versus Weighted Scores
| Approach | Strength | Weakness |
|---|---|---|
| Weighted score | Produces one continuous value | Weights can be arbitrary and difficult to interpret |
| Optimizing plus satisficing metrics | Closely represents hard requirements | Requires meaningful thresholds |
| Pareto analysis | Shows multiple tradeoffs | Does not automatically choose one model |
| Lexicographic ordering | Creates explicit priority levels | Can ignore meaningful improvements in lower-priority metrics |
Optimizing and satisficing metrics work especially well when some requirements naturally behave as constraints.
What If No Model Satisfies the Constraints?
If the feasible set is empty:\[ \mathcal{F}=\varnothing \]
then no candidate meets all requirements.
Possible responses include:
- Improve the models
- Compress or optimize inference
- Change the hardware
- Relax a threshold
- Redesign the product requirement
- Use different models for different operating conditions
The constraint should not be silently ignored. Instead, the conflict should be made explicit.
What If Many Models Satisfy the Constraints?
If many candidates satisfy every threshold, the optimizing metric provides a simple ranking.
However, models with nearly identical optimizing scores may still be compared using secondary considerations such as:
- Engineering complexity
- Training cost
- Interpretability
- Operational risk
- Maintenance burden
These can be introduced as additional constraints or used as tie-breakers.
Evaluation Sets and Metrics
These metrics must be calculated on appropriate data.
During development:\[ \text{Metrics are calculated on the development set} \]
For final evaluation:\[ \text{Metrics are calculated on the test set} \]
The development and test sets should represent the distribution on which the constraints and objective matter.
A latency constraint should also be measured under realistic hardware and workload conditions. A model that meets the threshold on a powerful development machine may fail it on the intended deployment device.
Avoid Moving Thresholds to Favor a Model
Thresholds should ideally be defined before comparing final candidates.
If a team repeatedly changes the latency limit from 100 ms to 120 ms and then 150 ms merely to admit a preferred model, the constraint no longer provides a stable target.
Thresholds can legitimately change when product requirements change, but the reason should be documented.
A Practical Selection Procedure
- List every important property.
- Identify the property that should be optimized continuously.
- Convert the remaining requirements into measurable thresholds.
- Define how and where each metric will be measured.
- Reject candidates that violate any threshold.
- Rank eligible candidates by the optimizing metric.
- Use operational considerations as documented tie-breakers.
- Revisit the formulation if the resulting model conflicts with real-world preferences.
Key Takeaway
When several metrics matter, choose one as the optimizing metric and make it as good as possible. Treat the others as satisficing metrics that must meet clearly defined thresholds. This produces a transparent and efficient way to compare models without forcing unrelated concerns into an arbitrary weighted sum.