A Basic Recipe for Improving Machine Learning Models
Training and development errors do more than describe a model’s performance. Together, they help diagnose whether the model has high bias, high variance, both problems, or neither.
Once the problem has been diagnosed, you can choose improvements that directly address it instead of experimenting randomly.
The Diagnostic Workflow
After training an initial model, ask two questions in order:
- Does the model have high bias?
- After reducing bias, does it have high variance?
The order matters because variance is most useful to evaluate after the model fits the training data reasonably well.
The overall process is:\[ \text{Train an initial model} \rightarrow \text{Check bias} \rightarrow \text{Reduce bias if necessary} \rightarrow \text{Check variance} \rightarrow \text{Reduce variance if necessary} \]
Repeat this cycle until both problems reach acceptable levels.
Step 1: Check for High Bias
To diagnose bias, examine the model’s performance on the training set.
A more precise analysis compares training error with the best achievable error:\[ \text{Avoidable bias} \approx \text{Training error} – \text{Optimal error} \]
When human-level performance is a reasonable approximation of optimal performance, it can be used as the reference point.
If the model performs poorly even on the training data, it is underfitting and likely has high bias.
High bias means that the model cannot fit the training data sufficiently well.
What to Try When Bias Is High
Several approaches can reduce high bias.
Use a larger neural network
Increase the model’s representational capacity by using:
- More hidden layers
- More hidden units per layer
- Both additional layers and units
A larger network can learn more complex functions and generally fits the training data more effectively.
Train the model longer
The model may be capable of fitting the data but may not have completed optimization.
Possible improvements include:
- Running more training iterations
- Improving the learning-rate configuration
- Using a more effective optimization algorithm
Training longer is useful when the cost is still decreasing. If optimization has already converged to an unsatisfactory result, additional time alone may not solve the problem.
Improve the optimization process
An advanced optimization method may reach a better solution faster or more reliably than basic gradient descent.
Optimization improvements are particularly relevant when the network has enough capacity but training fails to find parameters that fit the data well.
Try a more suitable architecture
A different neural-network architecture may better match the structure of the problem.
For example, different data types may benefit from architectures designed for:
- Images
- Sequences
- Audio
- Text
- Structured data
Architecture changes are less predictable than simply increasing capacity. They often require experimentation and domain knowledge.
Continue Until the Training Data Is Fit Well
When optimal error is reasonably low, a sufficiently large and well-optimized network should usually be able to fit the training data well.
The goal at this stage is not necessarily perfect generalization. The immediate goal is to remove the underfitting problem.
Resolve the bias problem before focusing heavily on variance.
There are exceptions. If the data is ambiguous, noisy, or inherently difficult, even the best model may have substantial error. For example, severely blurred images may be impossible for either a person or an algorithm to classify reliably.
That is why training error should be judged relative to achievable performance rather than always compared with zero.
Step 2: Check for High Variance
Once the model fits the training set reasonably well, examine its development-set performance.
Variance is reflected in the generalization gap:\[ \text{Generalization gap} = \text{Development error} – \text{Training error} \]
A large gap means that the model performs well on the training set but poorly on unseen development data.
High variance means that the model has learned the training data without generalizing effectively.
What to Try When Variance Is High
Obtain more training data
More data is often one of the most effective ways to reduce variance.
Additional examples make it more difficult for the model to memorize accidental patterns or noise. They encourage it to learn relationships that generalize across a broader population.
However, obtaining more labeled data may be difficult, expensive, or impossible.
Use regularization
Regularization discourages the model from relying too heavily on complex or unstable parameter configurations.
Common regularization techniques include:
- \(L_2\) regularization
- Dropout
- Data augmentation
- Early stopping
Regularization typically reduces overfitting, although sufficiently strong regularization can increase bias.
Try a more appropriate architecture
A better architecture may represent the important patterns more efficiently and generalize better.
Architecture selection can sometimes reduce both bias and variance, but it is difficult to prescribe a universally reliable method. The best choice depends on the structure of the data and the task.
Why More Data Does Not Usually Fix High Bias
Suppose a model cannot fit its current training data. Adding more examples from the same distribution generally does not increase its capacity or correct its optimization problem.
If the model is too simple, it may underfit both the original and expanded datasets.
Therefore, when bias is high, collecting more data is usually not the most efficient first response.
More data primarily addresses variance, not a model’s inability to fit its training data.
Instead, first consider:
- Increasing model capacity
- Training longer
- Improving optimization
- Choosing a more suitable architecture
Why a Larger Network Does Not Necessarily Increase Variance
Traditional machine-learning discussions often present model complexity as a strict trade-off:
- A more complex model reduces bias but increases variance.
- A simpler model reduces variance but increases bias.
This remains a useful conceptual framework, but the trade-off can be less restrictive in modern deep learning.
With adequate regularization, a larger neural network can reduce bias without substantially increasing variance. Likewise, more training data can reduce variance without meaningfully increasing bias.
This gives practitioners more independent tools:
| Objective | Common approach |
|---|---|
| Reduce bias | Use a larger network |
| Reduce variance | Obtain more data |
| Control variance in a large network | Apply regularization |
| Improve either problem | Explore a better architecture |
Why Deep Learning Weakens the Traditional Trade-Off
Modern deep learning benefits from two powerful resources:
- The ability to train large neural networks
- Access to large datasets
A larger, appropriately regularized model generally has enough capacity to reduce underfitting. More data generally improves its ability to generalize.
This means that it is sometimes possible to:
- Reduce bias without significantly increasing variance
- Reduce variance without significantly increasing bias
These options were less available in earlier machine-learning systems, where model capacity, data, and computation were more constrained.
The Cost of a Very Large Network
When regularization is applied appropriately, using a network larger than strictly necessary often does not damage predictive performance.
The primary cost may instead be computational:
- Longer training time
- Greater memory consumption
- Higher hardware requirements
- Slower experimentation
- More expensive deployment
A larger network is therefore not free, but its main disadvantage may be efficiency rather than increased variance.
Regularization and the Remaining Trade-Off
Regularization is primarily used to reduce variance, but it can introduce some bias.
As regularization becomes stronger, the model’s parameters are increasingly constrained. This can improve generalization, but excessive regularization may prevent the model from fitting the training data.
The desired level of regularization should:
- Reduce the development error
- Narrow the generalization gap
- Avoid causing a large increase in training error
A sufficiently large network often provides enough capacity that moderate regularization can reduce variance without creating a serious bias problem.
A Systematic Decision Process
The complete diagnostic process can be organized as follows.
First, evaluate bias
Compare training error with the optimal or human-level error.
If bias is high, try:
- A larger network
- Longer training
- Better optimization
- A more appropriate architecture
Then train and evaluate again.
Next, evaluate variance
Once training performance is satisfactory, compare development error with training error.
If variance is high, try:
- More training data
- Regularization
- A more suitable architecture
Then repeat the evaluation.
Finish when both are acceptable
A model is in a good state when:
- Its training error is reasonably close to the optimal error.
- Its development error is reasonably close to its training error.
- Its absolute development performance satisfies the application’s requirements.
Compact Decision Guide
| Diagnosis | Evidence | Useful next steps |
|---|---|---|
| High bias | Training error is far above optimal error | Larger network, longer training, better optimization, different architecture |
| High variance | Development error is far above training error | More data, regularization, different architecture |
| High bias and high variance | Poor training performance and a large generalization gap | Reduce bias first, then address variance |
| Low bias and low variance | Good training performance and a small generalization gap | Evaluate whether absolute performance meets the goal |
Important Assumptions
This process works most directly when:
- Training and development data come from the same distribution.
- The optimal error can be reasonably estimated.
- Training and development measurements are based on sufficiently large datasets.
- The selected error metric reflects the real objective.
If the distributions differ, the development gap may include distribution mismatch as well as variance. If the optimal error is high, a seemingly large training error may not indicate substantial bias.
Key Takeaway
Use diagnosis to determine what to try next.
If training performance is poor, focus on reducing bias.
Useful approaches include a larger network, longer training, improved optimization, or a better architecture.
If training performance is good but development performance is poor, focus on reducing variance.
Useful approaches include more data, regularization, or a more suitable architecture.
Modern deep learning reduces the severity of the traditional bias–variance trade-off. Large, well-regularized networks can reduce bias without necessarily causing excessive variance, while additional data can reduce variance without significantly worsening bias.