A Systematic Strategy for Improving Machine Learning Models

A model reaches 82% accuracy. You could collect more data, train longer, add regularization, change the architecture, fix labels, or add a feature. The next experiment should address a plausible source of error at a cost you can justify. Begin by defining a useful improvement, then compare training behavior and inspect representative mistakes before choosing what to try.

A decision rule for comparing models

Higher precision and lower recall create a tradeoff whose value depends on the application. One practical policy is to choose a primary metric and treat other requirements as satisficing constraints: conditions a candidate must meet. Multiple objectives can also be compared through costs or a Pareto frontier, the candidates for which no other candidate is at least as good on every objective and strictly better on one. What matters is an explicit selection rule. The example maximizes F1, which combines precision and recall, subject to latency and size limits; F1 is not automatically the right objective for every task.

def is_acceptable(metrics, constraints):
    ops = {"<=": lambda a, b: a <= b, ">=": lambda a, b: a >= b}
    return all(ops[op](metrics[k], t) for k, (op, t) in constraints.items())


def best_model(candidates, optimizing="f1", constraints=None, maximize=True):
    eligible = 
    if not eligible:
        return None
    choose = max if maximize else min
    return choose(eligible, key=lambda c: c[optimizing])


candidates = [
    {"name": "A", "f1": 0.91, "latency_ms": 340, "model_mb": 500},
    {"name": "B", "f1": 0.89, "latency_ms": 80,  "model_mb": 120},
    {"name": "C", "f1": 0.93, "latency_ms": 95,  "model_mb": 900},
]
constraints = {"latency_ms": ("<=", 100), "model_mb": ("<=", 500)}
winner = best_model(candidates, "f1", constraints)
print(winner["name"] if winner is not None else "No eligible model")
# B

Model C has the best F1 but exceeds the 500 MB size limit; A fails the latency limit. B meets both. The code assumes all required values are present and finite and returns the first candidate in an exact tie. Set maximize=False for a quantity such as loss. Before using measured latency as a constraint, define its workload, hardware, and statistic, such as the 95th percentile at a specified concurrency. A point estimate close to a limit needs repeated measurement. Candidates should use the same evaluation data and protocol.

If a metric conflicts with judgments on real examples, inspect the disagreements: the metric, labels, sample, or judgment may be at fault. Revise the evaluation policy when that investigation supports a change, document why, and rescore candidates consistently. Choosing a metric after seeing which model it favors can overfit the selection process. Use the development set (dev, or validation set) for iteration and reserve an untouched test set for the final assessment.

Learning curves and regularization experiments are discussed in Diagnosing and Preventing Overfitting in Deep Learning.

What a human baseline can tell you

Bayes error is the minimum expected classification error attainable from the available inputs and target distribution. It can be zero for some problems. Human performance can provide a useful reference for a perceptual task, but it need not be close to Bayes error and models can surpass it. For a predictor restricted to the same information, its true expected error is an upper bound on that minimum; a finite measured human error rate is only an estimate of the predictor’s error.

Choose a relevant baseline and measure it on representative examples under a stated labeling protocol. If experts see clinical history while the model receives only an image, their error rates do not isolate the model’s room for improvement from that image. With comparable inputs and labels, training error of 8% is seven percentage points above a 1% human reference and only half a point above a 7.5% reference. The smaller gap does not mean the model is finished: dev performance, uncertainty in the baseline, and the task’s requirements still matter.

Training and development gaps

A common diagnostic vocabulary calls the train–human gap “avoidable bias” and the dev–train gap “variance.” Here they are differences between observed error rates, not the bias and variance terms of a statistical decomposition. Compute training error in evaluation mode with the same preprocessing, prediction rule, and error metric used on dev. Dropout, augmentation, or a regularization term included only in the training loss can otherwise make the comparison misleading. Comparable data distributions and label quality are also needed to interpret the gaps.

\[g_{\mathrm{reference}}=e_{\mathrm{train}}-e_{\mathrm{human}},\qquad g_{\mathrm{generalization}}=e_{\mathrm{dev}}-e_{\mathrm{train}}\] Here each \(e\) is a measured error rate. The code accepts fractions and reports percentage-point differences: 0.08 − 0.01 = 0.07, or seven percentage points. It leaves the choice of intervention to the investigation that follows.

def gap_report(human_err, train_err, dev_err):
    return {
        "train_minus_human_pp": round(100 * (train_err - human_err), 2),
        "dev_minus_train_pp": round(100 * (dev_err - train_err), 2),
    }


print(gap_report(0.01, 0.08, 0.10))
print(gap_report(0.01, 0.02, 0.10))
# {'train_minus_human_pp': 7.0, 'dev_minus_train_pp': 2.0}
# {'train_minus_human_pp': 1.0, 'dev_minus_train_pp': 8.0}

Both examples have 10% dev error. In the first, inspect optimization progress, fitting capacity, feature information, and training-label quality. In the second, compare learning curves, targeted data collection, augmentation, and regularization. These are candidate experiments, not guaranteed cures. More data can help an underfitting model, and a larger model can sometimes improve generalization. Negative gaps are possible too; check sampling variation, baseline comparability, and evaluation settings before interpreting their signs.

A worse test score can reflect repeated selection on dev, distribution shift, label differences, sampling variation, or an evaluation bug. The gap alone does not identify the cause. If the test result drives further model or metric choices, that test set has entered the development process; a new independent final evaluation is then needed. A larger dev set may reduce sampling noise, but cannot by itself correct a mismatched population or a broken pipeline.

Error analysis

The gaps help choose what to inspect, but concrete interventions often come from reading examples. Compare predictions with the inputs, labels, and relevant context. A dog-breed classifier might fail on poor lighting, similar breeds, or an inconsistent annotation rule. These observations suggest hypotheses; “night scene” describes an error’s context and does not prove that darkness caused it.

Draw a representative sample of dev mistakes, for example 100 chosen uniformly from all mistakes. The code below uses one primary category assigned to each reviewed example. Its last column estimates how much of the current overall error falls in that category: overall error rate multiplied by the category’s share of sampled errors. A targeted or stratified sample needs weights reflecting its selection scheme. Keep ambiguous cases visible and expand the review if the sample is too small for the decision.

from collections import Counter


def error_inventory(categories, dev_error_rate):
    """One primary category per uniformly sampled dev error."""
    if not categories:
        raise ValueError("review at least one error")
    if not 0 <= dev_error_rate <= 1:
        raise ValueError("error rate must be between 0 and 1")
    counts = Counter(categories)
    n = len(categories)
    return [(category, k, k / n, dev_error_rate * k / n)
            for category, k in counts.most_common()]


sample = (["blurry"] * 8 + ["mislabeled"] * 6 + ["similar_breed"] * 43
          + ["occlusion"] * 12 + ["night"] * 31)
for category, count, share, contribution in error_inventory(sample, 0.10):
    print(f"{category:14s} {count:3d} share {share:.2f} contribution_pp {100*contribution:.1f}")
# similar_breed   43 share 0.43 contribution_pp 4.3
# night           31 share 0.31 contribution_pp 3.1
# occlusion       12 share 0.12 contribution_pp 1.2
# blurry           8 share 0.08 contribution_pp 0.8
# mislabeled       6 share 0.06 contribution_pp 0.6

At 10% overall error, the estimated blur contribution is 0.10 × 8/100 = 0.008, or 0.8 percentage points. That is a hypothetical gain if all errors in that category were fixed and nothing else changed, not a guaranteed or statistically rigorous upper bound from 100 reviewed errors. The two largest categories account for 74% of this sample. Actual gain depends on how many errors a change fixes, which new errors it introduces, and whether it affects other categories. Repairing an evaluation label also changes the score without necessarily improving predictions.

Start with categories that help the investigation and revise them when examples reveal something missing. Multiple tags are useful, but then count examples carrying each tag rather than treating a flat list of tags as independent errors. Category shares can overlap and their estimated gains cannot simply be added. For a combined intervention, count the union of affected examples. Revisit the inventory after a substantial change.

A category’s share of errors is different from its error rate. To estimate the night-scene error rate, count mistakes among all reviewed night scenes, including correct predictions. The error-only sample cannot supply that denominator. A rare but consequential failure may deserve attention even when its contribution to overall accuracy is small.

Use the inventory to scope a pilot. For night augmentation, record the current night-slice score and sample count, define a useful improvement, and check daytime performance and the overall constraints too. Compare the likely gain with labeling, engineering, and compute costs. A cheap partial blur fix can be a better next experiment than a costly attempt at the largest category. Counts alone cannot decide how to spend a week.

Incorrect labels

Random training-label noise can damage learning and can be memorized; there is no universal “few percent is harmless” threshold. Its effect depends on dataset size, class balance, noise mechanism, and training procedure. A small overall error rate may conceal severe corruption in a rare class. Systematic errors, such as a consistently misapplied annotation rule, can teach a repeatable wrong association. Audit a representative sample before deciding between targeted cleanup, broader relabeling, or changes to training.

Incorrect dev or test labels can distort both the score and the comparison between models. Comparing the overall label-error rate with the candidates’ score gap does not establish that their ranking is unknowable: it matters which labels are wrong and how the candidates predict those examples. Review influential disagreements using an independent labeling protocol, and report uncertainty when the available evidence does not distinguish candidates.

Use consistent target definitions and adjudication standards across evaluation sets, and version the corrections so all candidates can be rescored on the same labels. Consistency alone does not guarantee matching input distributions. Inspect a representative sample of agreements as well as disagreements with the model: disagreement-only review can overstate the overall label-error rate and miss labels that the model has also learned incorrectly. Keep test-label adjudication independent of model selection; repeatedly inspecting test failures to choose improvements consumes the test set’s role as an untouched assessment.

Distribution mismatch and transfer learning are treated separately in Data Mismatch, Transfer Learning, and Multitask Learning.

Running experiments that answer something

• Build a baseline you can inspect. Use a simple system to check the data path, scoring, and important slices before a costly search. The right initial budget depends on access to data and the cost of a valid evaluation.
• Design comparisons that isolate the question. A one-factor ablation can isolate an effect under a fixed setting. When two changes may interact, compare baseline, A only, B only, and A plus B; the joint gain need not equal the separate gains.
• State the hypothesis and decision rule before running. For example: night-time augmentation should improve night-slice accuracy by at least three percentage points while meeting a stated daytime-regression limit and latency constraint. Record the baseline, evaluation population, and budget alongside that target.
• Separate training variation from evaluation uncertainty. Compare candidates on the same dev examples and, where useful, matched seed runs. Seed repeats measure training randomness conditional on the data. They do not replace a large enough evaluation sample or checks across relevant periods and sites. Resampling evaluation examples jointly for both candidates and recomputing the metric difference can assess evaluation uncertainty; correlated examples need an appropriate group or time-based method.
• Track relevant slices alongside the overall score. Report slice sizes and uncertainty, especially for rare cases. Decide which regressions are unacceptable from task requirements. Repeated searches over many slices can also produce apparent improvements by chance.
• Keep an experiment record. Record data and label versions, split identifiers, configuration, seeds, code version, baseline differences, cost, and the decision. An inconclusive or negative result should say what was tested and what remains unresolved.

After a pilot, keep a change only when its measured benefit and cost support the decision rule, or revise the hypothesis and gather more evidence. If remaining gains are too small to justify further work, stop and evaluate the selected system on the reserved test set. The next useful step may be improving the measurement or data rather than fitting another model.

Compressing and serving the finished model is covered in Model Compression and Deployment: Distillation, Pruning, and Serving.

Calibration and uncertainty estimation are covered in Uncertainty Estimation and Model Calibration in Deep Learning.

Exercises

1. Constraints and the selection rule. Use models A, B, and C from the first code block. For this exercise, remove the model-size constraint and maximize F1 subject only to latency at most 100 ms. Which candidate wins? Repeat at 90 ms, then restore the 500 MB size limit with the 100 ms latency limit.

2. Error inventory. A uniform sample of 100 dev mistakes contains 43 similar-breed cases, 31 night scenes, 12 occlusions, 8 blurry inputs, and 6 suspected label errors. Each example has one primary category, and overall dev error is 10%. Estimate each category’s contribution in percentage points. Which contributions are below one point, and what else do you need before choosing a week’s work?

from collections import Counter

cats = Counter({"similar_breed": 43, "night": 31, "occlusion": 12,
                "blur": 8, "mislabeled": 6})
dev_err, n = 0.10, sum(cats.values())
for category, count in cats.most_common():
    print(f"{category:14s} estimated contribution {100*dev_err*count/n:.1f} pp")
# similar_breed  estimated contribution 4.3 pp
# night          estimated contribution 3.1 pp
# occlusion      estimated contribution 1.2 pp
# blur           estimated contribution 0.8 pp
# mislabeled     estimated contribution 0.6 pp

3. Same dev error, different clues. Model A has human-reference error 1%, train error 8%, and dev error 10%. Model B has the same human-reference and dev errors but train error 2%. Compute the two gaps and propose one investigation for each. Explain why the numbers do not guarantee opposite effects from collecting data or enlarging the model.

References

scikit-learn: multi-metric evaluation illustrates recording several metrics while choosing a primary refit criterion. Harutyunyan et al. (2020), Improving Generalization by Controlling Label-Noise Information in Neural Network Weights, studies memorization of label noise and its relationship to generalization.