Defining Human-Level Performance and Estimating Bayes Error

The phrase “human-level performance” can refer to several different benchmarks:

• An average untrained person
• A typical professional
• An experienced expert
• A team of experts reaching a consensus

The correct definition depends on why the benchmark is being used.

For bias–variance analysis, human-level performance is most useful as an approximation of Bayes error—the lowest error theoretically achievable on the task.

When estimating Bayes error, use the strongest credible human performance available rather than the performance of an average individual.

What Is Bayes Error?

Bayes error is the minimum possible error achievable by any prediction function under the true data distribution:\[ E_{\text{Bayes}} = \min_f P(f(X)\neq Y) \]

It represents irreducible error arising from factors such as:

• Ambiguous inputs
• Missing information
• Measurement noise
• Incorrect or inconsistent labels
• Overlapping class distributions
• Randomness in the outcome

Bayes error is not necessarily zero.

For a noisy audio recording, for example, the spoken words may be impossible to determine with certainty. Even an ideal classifier cannot recover information that is absent from the input.

Why Human Performance Helps

Bayes error is usually unknown. On tasks that humans perform well, strong human performance can provide a practical approximation.

If an expert system of human evaluators achieves an error rate of:\[ 0.5\% \]

then Bayes error cannot be greater than \(0.5\%\), because at least one decision process has already achieved that result:\[ E_{\text{Bayes}}\leq0.5\% \]

The true Bayes error could be lower, but \(0.5\%\) provides a useful working estimate.

An Illustrative Medical-Image Example

Suppose performance on a narrowly defined image-classification task is:

Evaluator	Error
Untrained individual	\(3.0\%\)
Typical specialist	\(1.0\%\)
Experienced specialist	\(0.7\%\)
Expert team reaching consensus	\(0.5\%\)

Which value should be called human-level error?

The answer depends on the purpose.

For estimating Bayes error

Use:\[ E_{\text{human}}\approx0.5\% \]

The expert team is the strongest demonstrated human process, so its performance gives the tightest available upper bound on Bayes error.

For comparing a product with an individual user

A different benchmark may be relevant. If the question is whether a system outperforms a typical individual specialist, then:\[ 1.0\% \]

may be the appropriate comparison.

“Human-level performance” is not one universal number. The benchmark must be defined according to the decision it supports.

Human-Level Performance for Two Different Purposes

Purpose	Appropriate benchmark
Estimate Bayes error	Best credible human or expert-team performance
Demonstrate superiority to a typical practitioner	Typical practitioner’s performance
Compare with an experienced expert	Experienced expert’s performance
Decide whether the model assists a workflow	Performance under that workflow’s real conditions

Confusing these purposes can lead to incorrect bias–variance diagnosis.

Avoidable Bias

Traditional bias analysis sometimes compares training error with zero:\[ \text{Bias estimate} \approx E_{\text{train}}-0 \]

This works reasonably well when Bayes error is nearly zero.

For noisy or ambiguous tasks, a more useful measure is avoidable bias:\[ E_{\text{avoidable bias}} \approx E_{\text{train}}-E_{\text{Bayes}} \]

This measures the gap between current training performance and the best performance believed to be possible.

The word “avoidable” matters because the Bayes-error component cannot be eliminated merely by improving the learning algorithm.

Variance

Variance can be estimated using the gap between development and training performance:\[ E_{\text{variance}} \approx E_{\text{dev}}-E_{\text{train}} \]

This measures how much performance degrades when moving from data used for parameter learning to unseen development data.

A large gap suggests the model does not generalize sufficiently well.

Example 1: Avoidable Bias Dominates

Suppose:\[ E_{\text{Bayes}}\approx0.5\% \]\[ E_{\text{train}}=5.0\% \]\[ E_{\text{dev}}=6.0\% \]

Avoidable bias is:\[ 5.0\%-0.5\%=4.5\% \]

Variance is:\[ 6.0\%-5.0\%=1.0\% \]

Component	Estimate
Avoidable bias	\(4.5\%\)
Variance	\(1.0\%\)

Avoidable bias is the larger problem.

Possible responses include:

• Increase model capacity
• Use a more suitable architecture
• Improve optimization
• Train longer
• Reduce excessive regularization
• Improve input features

In this case, whether Bayes error is estimated as \(1.0\%\), \(0.7\%\), or \(0.5\%\) does not substantially change the conclusion. Avoidable bias remains much larger than variance.

Example 2: Variance Dominates

Suppose:\[ E_{\text{Bayes}}\approx0.5\% \]\[ E_{\text{train}}=1.0\% \]\[ E_{\text{dev}}=5.0\% \]

Avoidable bias is:\[ 1.0\%-0.5\%=0.5\% \]

Variance is:\[ 5.0\%-1.0\%=4.0\% \]

Component	Estimate
Avoidable bias	\(0.5\%\)
Variance	\(4.0\%\)

The dominant problem is variance.

Possible responses include:

• Collect more training data
• Apply L2 regularization
• Use dropout
• Use data augmentation
• Select an architecture that generalizes better

Again, small differences in the human benchmark do not alter the diagnosis.

Example 3: Precise Bayes Error Matters

Now suppose:\[ E_{\text{train}}=0.7\% \]\[ E_{\text{dev}}=0.8\% \]

Using the expert-team estimate:\[ E_{\text{Bayes}}\approx0.5\% \]

Avoidable bias is:\[ 0.7\%-0.5\%=0.2\% \]

Variance is:\[ 0.8\%-0.7\%=0.1\% \]

Component	Estimate
Avoidable bias	\(0.2\%\)
Variance	\(0.1\%\)

Both may deserve attention, but avoidable bias is approximately twice as large.

If the experienced individual’s \(0.7\%\) error were incorrectly used as the Bayes estimate:\[ 0.7\%-0.7\%=0 \]

The analysis would suggest no avoidable bias, potentially causing the team to overlook opportunities to improve training performance.

Accurate estimation of Bayes error matters most when the model is already close to human-level performance.

Comparing the Three Cases

Bayes estimate	Training error	Development error	Avoidable bias	Variance	Main focus
\(0.5\%\)	\(5.0\%\)	\(6.0\%\)	\(4.5\%\)	\(1.0\%\)	Bias
\(0.5\%\)	\(1.0\%\)	\(5.0\%\)	\(0.5\%\)	\(4.0\%\)	Variance
\(0.5\%\)	\(0.7\%\)	\(0.8\%\)	\(0.2\%\)	\(0.1\%\)	Both, with more bias

The framework is:\[ E_{\text{Bayes}} \rightarrow E_{\text{train}} \rightarrow E_{\text{dev}} \]

The first gap estimates avoidable bias, and the second estimates variance.

Why Progress Slows Near Human Performance

When training error is far above human-level error, the diagnosis is usually clear.

For example:\[ E_{\text{human}}=0.5\% \]\[ E_{\text{train}}=5.0\% \]

The exact Bayes error could be \(0.5\%\), \(0.4\%\), or \(0.3\%\), but avoidable bias is clearly large.

When training error reaches \(0.7\%\), tiny differences in the Bayes-error estimate become strategically important.

At this stage:

• Remaining improvements are small.
• Measurement noise matters more.
• Human benchmarks require careful definition.
• Label errors become more significant.
• Bias and variance gaps become harder to distinguish.
• Larger evaluation sets may be required.

The model is improving, but identifying the next useful intervention becomes more difficult.

Problems Where Bayes Error Is Nearly Zero

For some clean recognition tasks, human error may be extremely small. In such cases:\[ E_{\text{Bayes}}\approx0 \]

Then:\[ E_{\text{avoidable bias}} \approx E_{\text{train}} \]

Comparing training error with zero works reasonably well.

However, this assumption should not be applied automatically to noisy tasks.

Problems with Nonzero Bayes Error

Bayes error may be substantial when:

• Audio is too noisy to understand
• Images are blurry or obstructed
• Medical measurements are incomplete
• Different labels are genuinely plausible
• Outcomes contain unavoidable randomness
• The available features do not determine the target

Suppose:\[ E_{\text{Bayes}}=10\% \]

and:\[ E_{\text{train}}=11\% \]

Comparing training error with zero would suggest an 11% bias problem. In reality, the estimated avoidable bias is only:\[ 11\%-10\%=1\% \]

This difference leads to a very different strategy.

Human Disagreement as Useful Information

When experts disagree, the disagreement may indicate:

• Ambiguous examples
• Inconsistent labeling standards
• Insufficient information
• Genuine uncertainty
• Differences in expertise

Consensus performance can provide a stronger benchmark, but disagreement itself also helps estimate task difficulty.

If even experienced experts cannot consistently agree, expecting the model to achieve zero error may be unrealistic.

The Benchmark Must Match the Model’s Information

A human benchmark is meaningful only when humans and the model receive comparable information.

If experts receive:

• Additional patient history
• Multiple images
• Higher-resolution inputs
• Follow-up outcomes

while the model receives only one low-resolution image, their performance is not directly comparable.

Likewise, if the model has access to large historical databases unavailable to humans, it may legitimately outperform an individual.

A fair comparison should specify:

• Available inputs
• Evaluation conditions
• Time limits
• Expert qualifications
• Whether collaboration is allowed
• The exact metric

Human Performance Is an Upper Bound on Bayes Error

If a human process achieves error \(h\), then:\[ E_{\text{Bayes}}\leq h \]

It does not prove:\[ E_{\text{Bayes}}=h \]

A better classifier—human or machine—might still exist.

Human-level error is therefore a proxy, not a mathematical determination of the true optimum.

The proxy is useful as long as it supports better decisions about bias and variance.

After Surpassing Human-Level Performance

If:\[ E_{\text{train}}<E_{\text{human}} \]

then human performance becomes a weaker estimate of Bayes error.

The true optimum could be:

• Slightly below the model
• Far below the model
• Close to the model’s training error

This makes avoidable bias difficult to estimate.

Variance remains more directly observable:\[ E_{\text{dev}}-E_{\text{train}} \]

but deciding whether further training-set improvement is possible becomes less clear.

Other tools become increasingly important:

• Better labels
• Expert adjudication
• Larger evaluation sets
• Statistical confidence intervals
• Targeted error analysis
• Robustness evaluation
• Stronger reference systems

A Practical Diagnostic Procedure

1. Identify all credible human-performance estimates.
2. Determine why human-level performance is being measured.
3. For Bayes-error estimation, choose the strongest credible benchmark.
4. Calculate:

\[ E_{\text{train}}-E_{\text{Bayes}} \]

5. Calculate:

\[ E_{\text{dev}}-E_{\text{train}} \]

6. Compare the two gaps.
7. Focus first on the larger source of error.
8. Reconsider the Bayes estimate as model performance approaches it.
9. Stop relying heavily on the human proxy after the model surpasses it.

Key Takeaway

Human-level performance should be defined according to its purpose. When it is used to estimate Bayes error, use the best credible human or expert-team performance available. The gap from Bayes error to training error estimates avoidable bias, while the gap from training error to development error estimates variance. This distinction becomes especially important—and increasingly difficult—as model performance approaches or surpasses the human benchmark.