Understanding Bias and Variance in Machine Learning
Bias and variance are fundamental concepts for diagnosing machine-learning models. They help explain whether a model is failing because it cannot fit the available data or because it does not generalize well to new data.
Although the basic definitions are easy to learn, applying them correctly requires careful comparison of training and development performance. The interpretation also depends on factors such as human-level performance, the theoretically optimal error, and whether the datasets come from the same distribution.
Underfitting, Overfitting, and a Good Fit
Consider a binary-classification problem with two input features, \(x_1\) and \(x_2\).
High bias: underfitting
A simple linear classifier may fail to capture a curved relationship in the data. If the true decision boundary is nonlinear, a straight line will not fit the training examples well.
This is called:
- High bias
- Underfitting
The model is too limited to represent the underlying pattern.
High variance: overfitting
At the other extreme, an excessively complex model may fit nearly every training example, including noise and unusual observations.
This is called:
- High variance
- Overfitting
The model performs very well on the training data but fails to generalize to new examples.
An appropriate fit
A model with an appropriate level of complexity captures the main structure of the data without fitting every irregularity.
A useful model should fit the training data well while continuing to perform well on unseen data.
Diagnosing High-Dimensional Models
With only two input features, it may be possible to plot the data and inspect the decision boundary visually.
Modern neural networks, however, often work with thousands or millions of features. Their decision boundaries cannot be visualized directly.
Instead, bias and variance are diagnosed primarily by comparing:
- Training-set error
- Development-set error
The training error indicates how well the model fits the data it learned from. The gap between training and development error indicates how well it generalizes.
Assumptions Behind the Basic Analysis
The simplest bias-and-variance analysis assumes that:
- The best achievable error is very low.
- The training and development sets come from the same distribution.
For example, people can generally identify clear pictures of cats with nearly perfect accuracy. Human-level error—and likely the theoretically optimal error—is therefore close to zero for a clean version of this task.
Under these assumptions:
- Training error substantially above zero suggests avoidable bias.
- Development error substantially above training error suggests variance.
These assumptions matter because the same numerical errors may have very different meanings in a task where even the best possible classifier cannot perform well.
Example 1: High Variance
Suppose a cat classifier produces:
| Dataset | Error |
|---|---|
| Training | 1% |
| Development | 11% |
The model performs very well on the training set but much worse on the development set.
The generalization gap is:\[ 11\%-1\%=10\% \]
Because the model fits the training data well, high bias is not the main concern. The large increase on the development set suggests that the model has overfit the training data.
Low training error combined with much higher development error indicates high variance.
Example 2: High Bias
Consider another result:
| Dataset | Error |
|---|---|
| Training | 15% |
| Development | 16% |
Assuming the optimal error is close to zero, a 15% training error is poor. The model cannot fit even the data on which it was trained.
The generalization gap is only:\[ 16\%-15\%=1\% \]
This suggests that variance is relatively low. The primary problem is high bias.
High training error indicates that the model is underfitting the training data.
Example 3: High Bias and High Variance
Now consider:
| Dataset | Error |
|---|---|
| Training | 15% |
| Development | 30% |
The training error is already high, indicating high bias. The development error is another 15 percentage points worse:\[ 30\%-15\%=15\% \]
That large gap also indicates high variance.
The model therefore suffers from both problems:
- It does not fit the training set well.
- It does not generalize well from the training set to the development set.
This is effectively the worst of both worlds.
Example 4: Low Bias and Low Variance
Suppose the results are:
| Dataset | Error |
|---|---|
| Training | 0.5% |
| Development | 1% |
The training error is close to the assumed optimal error, so the model has low bias.
The generalization gap is also small:\[ 1\%-0.5\%=0.5\% \]
The model therefore has low variance as well.
If a 1% development error is satisfactory for the application, this would be considered a strong model.
Summary of the Four Cases
Assuming that the optimal error is approximately zero:
| Training error | Development error | Diagnosis |
|---|---|---|
| Low | Much higher | High variance |
| High | Slightly higher | High bias |
| High | Much higher | High bias and high variance |
| Low | Slightly higher | Low bias and low variance |
The basic logic is:
Compare training error with the best achievable error to diagnose bias.
Compare development error with training error to diagnose variance.
The Role of Bayes Error
The Bayes error is the lowest error theoretically achievable for a task. It reflects unavoidable uncertainty or ambiguity in the data.
In practice, human-level performance is sometimes used as an approximation of Bayes error, particularly for tasks that people perform well.
Suppose the optimal error is approximately:\[ 15\% \]
and a model produces:
| Dataset | Error |
|---|---|
| Training | 15% |
| Development | 16% |
It would be incorrect to diagnose high bias merely because the training error is 15%. If no classifier can do much better than 15%, the model is already close to optimal on the training set.
Its avoidable bias is approximately:\[ 15\%-15\%=0\% \]
Its generalization gap is:\[ 16\%-15\%=1\% \]
This model would have low avoidable bias and relatively low variance.
Avoidable Bias
A more accurate way to measure bias is to compare training error with the optimal error:\[ \text{Avoidable bias} \approx \text{Training error} – \text{Optimal error} \]
For example, if:\[ \text{Optimal error}=2\% \]
and:\[ \text{Training error}=8\% \]
then:\[ \text{Avoidable bias}\approx 8\%-2\%=6\% \]
The model has significant room to improve its fit to the training distribution.
By contrast, if:\[ \text{Optimal error}=15\% \]
and:\[ \text{Training error}=15.5\% \]
then:\[ \text{Avoidable bias}\approx0.5\% \]
The raw training error looks high, but the model is already close to the best possible performance.
Measuring Variance
Variance is commonly estimated using the difference between development and training error:\[ \text{Variance indicator} \approx \text{Development error} – \text{Training error} \]
For example:\[ \text{Training error}=2\% \]\[ \text{Development error}=12\% \]
gives:\[ \text{Generalization gap}=10\% \]
This large gap suggests that the model is highly sensitive to the training data and does not generalize effectively.
A More Complete Diagnostic Framework
The three relevant performance levels are:\[ \text{Optimal error} \rightarrow \text{Training error} \rightarrow \text{Development error} \]
Their differences answer different questions.
Optimal error to training error
\[ \text{Training error}-\text{Optimal error} \]
This estimates avoidable bias.
Training error to development error
\[ \text{Development error}-\text{Training error} \]
This estimates variance or the generalization gap.
This creates a more precise diagnosis than looking at either error in isolation.
Why Distribution Matching Matters
The simple variance interpretation assumes that the training and development sets come from the same distribution:\[ P_{\text{train}}(x,y) = P_{\text{dev}}(x,y) \]
If the distributions differ, a higher development error may be caused by two separate factors:
- The model may be overfitting.
- The development data may be inherently different from the training data.
For example, suppose the training set contains high-resolution professional photographs while the development set contains blurry mobile-phone images. A performance decline may reflect distribution mismatch rather than variance alone.
More detailed analysis is needed when the training and development distributions differ.
A Model Can Have Both High Bias and High Variance
Bias and variance are not mutually exclusive. A classifier may underfit the overall structure while overfitting specific parts of the data.
For example, imagine a classifier whose decision boundary is mostly linear even though the correct relationship is curved. This creates high bias.
However, suppose the boundary also bends sharply around a few unusual or mislabeled examples. This local flexibility indicates high variance.
The model then has:
- High bias because its overall structure is too simple
- High variance because some regions respond too strongly to individual examples
Such behavior may appear artificial in a two-dimensional illustration, but it can occur naturally in high-dimensional models. Different parts of the input space may exhibit different combinations of underfitting and overfitting.
Why the Traditional Trade-Off Is Less Restrictive in Deep Learning
In traditional machine learning, bias and variance were often described as a strict trade-off:
- Increasing model complexity reduced bias but increased variance.
- Simplifying the model reduced variance but increased bias.
This trade-off still exists conceptually, but modern deep learning provides more ways to address the two problems separately.
For example:
- A larger network can reduce bias.
- More data can reduce variance.
- Regularization can reduce variance.
- Better optimization can reduce bias.
- Improved architecture can reduce bias without necessarily causing unacceptable variance.
As a result, deep-learning practitioners still diagnose bias and variance, but they do not always assume that improving one must significantly worsen the other.
Practical Diagnostic Questions
When evaluating a model, ask these questions in order.
1. What is the best achievable performance?
Estimate the Bayes error or use human-level performance as a practical reference when appropriate.
2. How well does the model fit the training data?
Compare training error with the best achievable error.
A large gap suggests high avoidable bias.
3. How well does the model generalize?
Compare development error with training error.
A large gap suggests high variance.
4. Do the datasets come from the same distribution?
If they do not, avoid attributing the entire generalization gap to variance.
Key Takeaway
Bias and variance can be diagnosed using three reference points:\[ \text{Optimal error} \rightarrow \text{Training error} \rightarrow \text{Development error} \]
The first gap measures avoidable bias:\[ \text{Training error}-\text{Optimal error} \]
The second gap measures the model’s generalization problem:\[ \text{Development error}-\text{Training error} \]
High training error relative to the optimal error suggests high bias.
Development error substantially above training error suggests high variance.
This analysis works most directly when the optimal error is low and the training and development sets come from the same distribution. When these assumptions do not hold, the interpretation must account for irreducible error and distribution mismatch.