Error Analysis: Prioritizing Machine Learning Improvements
When a model performs worse than expected, there may be dozens of plausible improvements:
- Collect more data
- Add model capacity
- Improve blurry-image handling
- Add examples from difficult classes
- Correct label problems
- Engineer a new feature
- Change the architecture
Implementing each idea may require days or months. Error analysis helps estimate which directions have the greatest potential before committing substantial resources.
Manually inspect a sample of development-set errors, categorize them, and use their frequencies to estimate the maximum benefit of addressing each category.
Why Manual Error Analysis Helps
Suppose an image classifier has:\[ 90\% \text{ accuracy} \]
or equivalently:\[ 10\% \text{ error} \]
A team member notices that the model sometimes classifies dogs as cats and suggests a project specifically targeting dog images.
Possible actions include:
- Collecting more dog images
- Adding difficult negative examples
- Designing dog-specific features
- Changing the loss function
- Training a specialized model
- Modifying the architecture
Before beginning a large project, the team should estimate how much of the total error is actually caused by dogs.
Start with Misclassified Development Examples
Select approximately 100 misclassified examples from the development set and inspect them manually.
For each example, ask:
- Is this a dog?
- Is it a large wild cat?
- Is the image blurry?
- Is the label incorrect?
- Does the image contain a filter?
- Is it an unusual viewpoint?
- Is there another recurring pattern?
The sample does not have to contain exactly 100 examples, but 100 makes percentages easy to interpret and can usually be reviewed quickly.
The inspected examples should include both:
- False positives
- False negatives
Scenario 1: Dogs Cause Few Errors
Suppose five of the 100 examined mistakes involve dogs:\[ \frac{5}{100}=5\% \]
This means dog-related cases represent approximately 5% of the model’s current errors.
If the current overall error is:\[ 10\% \]
and every dog-related error could be eliminated, the estimated maximum absolute improvement is:\[ 10\%\times5\%=0.5\% \]
The error could decrease from:\[ 10\% \]
to approximately:\[ 9.5\% \]
This is a ceiling estimate. A real intervention would probably eliminate only some dog-related mistakes, so the actual improvement would likely be smaller.
Scenario 2: Dogs Cause Many Errors
Now suppose 50 of the 100 examined errors involve dogs:\[ \frac{50}{100}=50\% \]
The maximum absolute improvement would be:\[ 10\%\times50\%=5\% \]
If all dog-related errors were eliminated, overall error could theoretically decrease from:\[ 10\% \]
to:\[ 5\% \]
A project that could potentially halve the error rate may justify substantial effort.
Performance Ceiling
Suppose category \(k\) accounts for fraction \(q_k\) of the observed errors, and the current overall error is \(E\).
If every error in that category were eliminated, the approximate maximum absolute improvement would be:\[ \Delta E_{k,\max} = E q_k \]
The best possible remaining error would be:\[ E_{\min,k} = E(1-q_k) \]
This is called the category’s performance ceiling.
Example
If:\[ E=10\% \]
and:\[ q_k=0.08 \]
then:\[ \Delta E_{k,\max} = 10\%\times0.08 = 0.8\% \]
The best-case error after solving that category would be:\[ 10\%-0.8\%=9.2\% \]
Relative Versus Absolute Improvement
If error decreases from 10% to 9.5%, the absolute reduction is:\[ 10\%-9.5\%=0.5 \text{ percentage points} \]
The relative reduction is:\[ \frac{10\%-9.5\%}{10\%} = 5\% \]
These should not be confused.
| Description | Value |
|---|---|
| Original error | \(10\%\) |
| New error | \(9.5\%\) |
| Absolute reduction | \(0.5\) percentage points |
| Relative error reduction | \(5\%\) |
Clear terminology prevents small absolute changes from being mistaken for large changes in accuracy.
Evaluating Several Ideas at Once
Error analysis becomes even more useful when several possible projects are being considered.
Suppose the proposed categories are:
- Dogs mistaken for cats
- Large wild cats mistaken for domestic cats
- Blurry images
- Stylized image filters
Create a table with one row per misclassified example and one column per category.
| Example | Dog | Large cat | Blurry | Filter | Comment |
|---|---|---|---|---|---|
| 1 | ✓ | Pit bull | |||
| 2 | ✓ | Motion blur | |||
| 3 | ✓ | ✓ | Lion photographed in rain | ||
| 4 | ✓ | Strong color filter | |||
| 5 | ✓ | ✓ | Dog in low light |
One example can belong to multiple categories.
For example, an image may be:
- A large cat
- Blurry
- Taken in low light
- Incorrectly labeled
Therefore, column percentages do not need to sum to 100%.
Example Results
After inspecting 100 mistakes, suppose the results are:
| Error category | Number | Percentage |
|---|---|---|
| Dogs | 8 | \(8\%\) |
| Large wild cats | 43 | \(43\%\) |
| Blurry images | 61 | \(61\%\) |
| Stylized filters | 12 | \(12\%\) |
These values suggest that blurry images and large-cat examples have greater potential than dog or filter-related errors.
However, frequency alone does not dictate what the team must do. It estimates the available opportunity.
A high-frequency category may still be:
- Technically difficult
- Expensive to address
- Impossible with the available input
- Less important to users
- Already near its own Bayes limit
Categories Can Overlap
Suppose 61% of errors involve blur and 43% involve large cats. It would be incorrect to assume that solving both categories eliminates:\[ 61\%+43\%=104\% \]
of the errors.
Some examples may belong to both categories.
For two categories \(A\) and \(B\):\[ P(A\cup B) = P(A)+P(B)-P(A\cap B) \]
The combined ceiling depends on the union of the categories, not the sum of their individual percentages.
This is another reason to record categories at the individual-example level.
Add Categories During the Review
The initial table does not need to anticipate every error type.
While reviewing examples, the team may discover recurring patterns such as:
- Strong image filters
- Nighttime photographs
- Unusual camera angles
- Occluded subjects
- Multiple animals
- Watermarks
- Incorrect crops
- Label errors
Add new columns as these categories become apparent. If necessary, revisit earlier examples and mark whether they belong to the newly discovered category.
Error analysis is partly measurement and partly discovery.
Include Comments
A comment column preserves qualitative observations that may later suggest solutions.
Useful notes include:
- “Pit bull resembles a cat from this angle”
- “Lion behind wet glass”
- “Object occupies less than 5% of image”
- “Label appears incorrect”
- “Heavy color filter”
- “Only tail is visible”
Percentages identify large categories, while comments help explain what makes those categories difficult.
Prioritization Requires More Than Frequency
A useful project-selection decision considers:\[ \text{Expected value} \approx \text{Error frequency} \times \text{Fraction solvable} \times \text{Importance} \]
and compares it with:\[ \text{Engineering cost} \]
A practical prioritization table might be:
| Category | Error share | Estimated solvability | User importance | Effort | Priority |
|---|---|---|---|---|---|
| Dogs | Low | High | Medium | Medium | Low |
| Large cats | High | Medium | High | Medium | High |
| Blur | Very high | Medium | High | High | High |
| Filters | Low | High | Low | Low | Medium |
This prevents the team from automatically choosing the largest category when a smaller category is much easier or more important.
Error Analysis Does Not Prove Causation
If 61% of mistakes are blurry, this does not prove blur caused every one of those errors.
Some examples might remain difficult even if the blur were removed. The model may also fail because:
- The object is too small
- The label is wrong
- The class is rare
- The background is misleading
The category percentage is a prioritization estimate, not a causal proof.
Choosing the Sample Size
A sample of 100 errors is often enough to reveal major patterns. Larger samples provide more stable estimates.
A useful approach is:
- Begin with 100 errors.
- Create and refine categories.
- Continue if the results remain uncertain.
- Stop when the leading categories are clear enough to guide action.
If a category appears in only one or two examples, a larger review may be needed before making a major investment.
Sample Errors Randomly
The inspected errors should be selected systematically.
If reviewers choose only memorable mistakes, the estimated frequencies may be distorted.
Prefer:
- Randomly sampled development errors
- A predefined time window
- Balanced samples of false positives and false negatives when both matter
- Stratified samples when important groups are rare
If stratified sampling is used, apply the appropriate weights before estimating overall frequencies.
False Positives and False Negatives
It is often useful to analyze these separately.
False positives
The model predicts the positive class when it is absent:\[ \hat{y}=1,\quad y=0 \]
Possible categories include:
- Similar-looking objects
- Background patterns
- Label errors
- Threshold problems
False negatives
The model misses a real positive example:\[ \hat{y}=0,\quad y=1 \]
Possible categories include:
- Small objects
- Occlusion
- Low contrast
- Unusual poses
- Poor image quality
The two error types may require different interventions.
Error Analysis Spreadsheet Structure
A practical spreadsheet can include:
| Column | Purpose |
|---|---|
| Example ID | Identify the input |
| True label | Reference label |
| Predicted label | Model output |
| Confidence | Prediction probability |
| Error type | False positive or false negative |
| Category columns | Mark recurring patterns |
| Label correct? | Flag suspected annotation problems |
| Comments | Record observations |
| Proposed action | Capture ideas |
| Reviewer | Track who inspected it |
This structure makes the analysis repeatable and shareable.
From Analysis to Action
After completing the review:
- Calculate the frequency of every category.
- Estimate each category’s ceiling.
- Estimate how much of that category is realistically solvable.
- Consider importance and engineering cost.
- Select one or more high-value directions.
- Design a targeted experiment.
- Measure whether the category-specific error decreases.
- Check whether overall development performance improves.
- Repeat error analysis after major changes.
Error categories often change as the model improves. Solving one large category may reveal another that was previously hidden.
Example of Targeted Follow-Up
Suppose blurry images account for 61% of errors.
Possible interventions include:
- Collect more blurry training images
- Apply realistic blur augmentation
- Improve image preprocessing
- Increase input resolution
- Detect unusably blurry inputs
- Ask users to retake images
- Use a model architecture better suited to low-quality inputs
The error analysis identifies the problem category. It does not automatically determine the correct solution.
Compare Before and After
Maintain the same categories when evaluating a new model:
| Category | Baseline errors | New model errors | Change |
|---|---|---|---|
| Dogs | 8 | 7 | Small improvement |
| Large cats | 43 | 25 | Strong improvement |
| Blur | 61 | 55 | Small improvement |
| Filters | 12 | 13 | Slight regression |
This reveals whether the intervention produced the intended effect and whether it introduced new regressions.
Common Mistakes
Starting a large project after seeing a few examples
A memorable pattern may account for only a small fraction of total error.
Analyzing training errors instead of development errors
Training errors can be useful, but development errors more directly reveal generalization failures affecting model selection.
Treating overlapping categories as independent
Adding category percentages can greatly exaggerate potential improvement.
Assuming the ceiling is achievable
Eliminating every error in a category is usually unrealistic.
Ignoring label mistakes
Some apparent model failures may be incorrect reference labels.
Keeping only quantitative totals
Qualitative comments often generate the most useful ideas.
Never repeating the analysis
The error distribution changes as the model improves.
Key Takeaway
Before investing heavily in an improvement idea, manually inspect a representative sample of development-set mistakes. Categorize the errors, estimate each category’s performance ceiling, and compare that opportunity with the likely difficulty and importance of solving it. A short, structured error analysis can prevent months of work on a problem that accounts for only a small fraction of total error.