Improving Model Performance with Avoidable Bias and Variance

A supervised-learning system must succeed at two fundamental tasks:

1. Fit the training distribution sufficiently well.
2. Generalize from the training set to unseen development data.

The first is primarily an avoidable-bias problem. The second is primarily a variance problem.

These problems should be diagnosed separately because they usually require different solutions.

Compare training error with Bayes error to estimate avoidable bias. Compare development error with training error to estimate variance.

The Three Error Levels

A useful diagnostic framework compares three quantities:\[ E_{\text{Bayes}} \rightarrow E_{\text{train}} \rightarrow E_{\text{dev}} \]

where:

• \(E_{\text{Bayes}}\) is the estimated irreducible error
• \(E_{\text{train}}\) is the training error
• \(E_{\text{dev}}\) is the development error

The two gaps reveal different weaknesses.

Avoidable bias

\[ E_{\text{avoidable bias}} \approx E_{\text{train}}-E_{\text{Bayes}} \]

This estimates how much better the model could realistically perform on the training distribution.

Variance

\[ E_{\text{variance}} \approx E_{\text{dev}}-E_{\text{train}} \]

This estimates how much performance is lost when moving from training data to unseen development data.

Why Bayes Error Matters

Training error should not always be compared with zero.

Suppose:\[ E_{\text{train}}=8\% \]

If Bayes error is:\[ E_{\text{Bayes}}=1\% \]

then:\[ E_{\text{avoidable bias}}=7\% \]

The model has considerable room to improve.

If Bayes error is instead:\[ E_{\text{Bayes}}=7.5\% \]

then:\[ E_{\text{avoidable bias}}=0.5\% \]

The same 8% training error is now close to the estimated optimum.

The appropriate strategy depends on the gap from Bayes error, not merely the absolute training error.

Human-Level Performance as a Proxy

For tasks humans perform well, strong human-level performance can approximate Bayes error:\[ E_{\text{Bayes}} \approx E_{\text{human}} \]

The strongest credible human benchmark is usually the most useful reference. This could be the performance of:

• An experienced individual
• A group of experts
• An expert consensus process
• A human assisted by specialized tools

This approximation becomes less reliable once the model surpasses the best available human benchmark.

Orthogonalizing Model Improvement

Orthogonalization means using different controls for different problems.

For machine learning:

• Bias controls should primarily improve training performance.
• Variance controls should primarily improve generalization.
• Evaluation controls should align model selection with real-world goals.

Perfect separation is impossible, but this framework makes experimentation more systematic.

Diagnosing Avoidable Bias

Avoidable bias is large when:\[ E_{\text{train}}-E_{\text{Bayes}} \]

is large.

This means the model is not fitting the training distribution nearly as well as appears possible.

Possible causes include:

• Insufficient model capacity
• Weak architecture
• Poor optimization
• Too little training
• Excessive regularization
• Poor initialization
• Inadequate input representation
• Implementation errors

Reducing Avoidable Bias

Train a larger model

Increase capacity by changing:

• Number of hidden units
• Number of layers
• Width of important layers
• Representational components

A larger model can represent more complex functions and may fit the training data more successfully.

Train longer

If the training loss is still decreasing, additional optimization steps may reduce avoidable bias.

Training longer is useful when the model has sufficient capacity but has not yet reached a good solution.

Improve the optimizer

Possible choices include:

• Gradient descent with momentum
• RMSprop
• Adam
• Better learning-rate schedules

Optimization improvements can help the model reach a lower training loss more quickly and reliably.

Tune the learning rate

A poor learning rate can prevent successful optimization:

• Too small: training progresses slowly.
• Too large: training oscillates or diverges.

The learning rate is often one of the most important hyperparameters to tune.

Improve the architecture

A more suitable architecture can represent the target relationship more efficiently.

Examples include:

• Convolutional structures for spatial data
• Recurrent or sequence models for ordered data
• Attention-based architectures
• Better activation functions
• Improved normalization

Architecture search is less predictable than simply increasing capacity, but it can produce major improvements.

Reduce excessive regularization

If regularization is too strong, the model may be prevented from fitting the training set.

Possible adjustments include:

• Reduce the L2 coefficient
• Increase dropout keep probability
• Remove unnecessary constraints
• Train longer under regularization

Verify the implementation

Large training error may come from:

• Incorrect labels
• Shape errors
• Wrong loss implementation
• Broken data preprocessing
• Gradient bugs
• Unintended frozen parameters

Before increasing model complexity, confirm that the existing system is functioning correctly.

Diagnosing Variance

Variance is large when:\[ E_{\text{dev}}-E_{\text{train}} \]

is large.

This means the model performs well on examples used for learning but poorly on unseen data.

Possible causes include:

• Too little training data
• Excessive model flexibility
• Weak regularization
• Limited data diversity
• Distribution mismatch
• Duplicate or unrepresentative examples
• Hyperparameter overfitting to the development set

Reducing Variance

Collect more training data

Additional representative data can reduce the model’s dependence on accidental patterns in the original training set.

The most useful data usually resembles the examples on which the system needs to generalize.

Apply L2 regularization

L2 regularization adds a weight penalty:\[ J_{\text{reg}} = J + \frac{\lambda}{2m} \sum_l \left\|W^{[l]}\right\|_F^2 \]

This discourages unnecessarily large weights and can reduce overfitting.

Use dropout

Dropout randomly removes activations during training, forcing the network to avoid relying excessively on individual units.

It can be effective but also changes optimization behavior, so it should be introduced deliberately.

Use data augmentation

Data augmentation creates transformed training examples that preserve the correct label.

For images, examples include:

• Horizontal flips
• Crops
• Translations
• Small rotations
• Color transformations

The transformations should reflect variations expected in the target application.

Find a better architecture

An architecture with appropriate inductive biases may generalize better even when it has substantial capacity.

For example, a convolutional model exploits spatial structure more effectively than an arbitrary fully connected network.

Review the data distribution

If training and development sets come from different distributions, the development gap may not represent ordinary variance.

In that case, more training data from the wrong distribution may not solve the problem. The team may need:

• More target-distribution data
• A revised development set
• Domain adaptation
• Better preprocessing
• A separate analysis of distribution mismatch

Decision Table

Diagnostic result	Likely problem	Priorities
Large Bayes-to-training gap	Avoidable bias	Larger model, longer training, better optimization
Large training-to-development gap	Variance	More data, regularization, augmentation
Both gaps are large	Bias and variance	Reduce bias first, then address variance
Both gaps are small	Near current target	Improve labels, features, metric, or data quality
Good offline results but poor production results	Misaligned target	Change metric or evaluation distribution

Example: Avoidable Bias Dominates

Suppose:\[ E_{\text{Bayes}}=1\% \]\[ E_{\text{train}}=8\% \]\[ E_{\text{dev}}=10\% \]

Then:\[ E_{\text{avoidable bias}}=7\% \]\[ E_{\text{variance}}=2\% \]

The larger opportunity is reducing avoidable bias.

Reasonable experiments include:

• Increase model size
• Train longer
• Tune the learning rate
• Use Adam or momentum
• Reduce excessive regularization

Collecting much more data may not be the most efficient first action.

Example: Variance Dominates

Suppose:\[ E_{\text{Bayes}}=7.5\% \]\[ E_{\text{train}}=8\% \]\[ E_{\text{dev}}=10\% \]

Then:\[ E_{\text{avoidable bias}}=0.5\% \]\[ E_{\text{variance}}=2\% \]

The training result is already close to the estimated limit, while development performance is substantially worse.

Reasonable actions include:

• Collect more representative data
• Add L2 regularization
• Use dropout
• Apply data augmentation
• Improve distribution matching

Example: Both Problems Are Large

Suppose:\[ E_{\text{Bayes}}=1\% \]\[ E_{\text{train}}=7\% \]\[ E_{\text{dev}}=14\% \]

Then:\[ E_{\text{avoidable bias}}=6\% \]\[ E_{\text{variance}}=7\% \]

Both problems are substantial.

A practical sequence is:

1. Increase capacity or improve optimization until training performance improves.
2. Monitor whether the development gap grows.
3. Add data or regularization to improve generalization.
4. Repeat until both gaps are acceptable.

Reducing bias first ensures that the model can represent and learn the task before restricting it for generalization.

Example: Neither Problem Is Large

Suppose:\[ E_{\text{Bayes}}=1.0\% \]\[ E_{\text{train}}=1.2\% \]\[ E_{\text{dev}}=1.4\% \]

Then:\[ E_{\text{avoidable bias}}=0.2\% \]\[ E_{\text{variance}}=0.2\% \]

The model may already be close to the current target.

Further progress may require:

• Better labels
• Additional input information
• More precise Bayes-error estimation
• Targeted analysis of rare failures
• Larger evaluation sets
• A better metric
• A revised application objective

Training Error Is Not the Entire Goal

A model can achieve extremely low training error through memorization without improving its population performance.

The objective is not simply:\[ E_{\text{train}}\rightarrow0 \]

It is:\[ E_{\text{train}} \rightarrow E_{\text{Bayes}} \]

while also achieving:\[ E_{\text{dev}} \approx E_{\text{train}} \]

The ideal pattern is:\[ E_{\text{Bayes}} \approx E_{\text{train}} \approx E_{\text{dev}} \]

A Systematic Improvement Process

1. Define the target distribution and evaluation metric.
2. Estimate Bayes error using the best available evidence.
3. Measure training error.
4. Measure development error.
5. Calculate the avoidable-bias gap.
6. Calculate the variance gap.
7. Identify the larger bottleneck.
8. Select experiments that primarily address that bottleneck.
9. Change a controlled number of variables.
10. Evaluate again and repeat.

Key Takeaway

Improve a supervised-learning system by separating its two central challenges: fitting the training distribution and generalizing to unseen data. The gap from Bayes error to training error measures avoidable bias, while the gap from training error to development error measures variance. Diagnose the larger gap first, then apply the controls designed specifically for that problem.