Relative accuracy

Relative accuracy expresses a model's performance as a ratio to a baseline's: above 1 is better than the baseline, below 1 is worse. It is the natural way to answer “how much did this help?” rather than “how good is it?”

The idea is right and the usual arithmetic is misleading, for a reason that is easy to see once measured.

Accuracy has a floor

import numpy as np

base, model = 0.70, 0.84
print(f"  baseline accuracy {base}, model accuracy {model}")
print(f"    accuracy ratio        {model / base:>8.4f}   'twenty percent better'")
print(f"    error rate            {1 - base:.2f} -> {1 - model:.2f}")
print(f"    error reduction       {1 - (1 - model) / (1 - base):>8.2%}   the same two numbers")
"""Accuracy has a floor set by the majority class, so a ratio of accuracies
   compresses every improvement toward 1 and hides how much was available."""

print(f"\n{'majority class':>16}{'baseline':>11}{'model':>9}{'accuracy ratio':>17}{'error reduction':>18}")
for maj in (0.50, 0.70, 0.90, 0.99):
    m = maj + (1 - maj) * 0.5          # the model closes half the available gap
    print(f"{maj:>16.2f}{maj:>11.2f}{m:>9.4f}{m / maj:>17.4f}{1 - (1 - m) / (1 - maj):>18.2%}")

for maj, m in ((0.99, 0.995), (0.50, 0.60)):
    print(f"    majority {maj:.2f}, model {m:.3f}: ratio {m / maj:.4f}"
          f"   error reduction {1 - (1 - m) / (1 - maj):.1%}")

for name, b, m in (("majority class", 0.70, 0.84), ("last year's model", 0.82, 0.84),
                   ("random guessing", 0.50, 0.84)):
    print(f"    against {name:>18}: ratio {m / b:.4f}   error reduction "
          f"{1 - (1 - m) / (1 - b):>7.1%}")
#   baseline accuracy 0.7, model accuracy 0.84
#     accuracy ratio          1.2000   'twenty percent better'
#     error rate            0.30 -> 0.16
#     error reduction         46.67%   the same two numbers
#
#   majority class   baseline    model   accuracy ratio   error reduction
#             0.50       0.50   0.7500           1.5000            50.00%
#             0.70       0.70   0.8500           1.2143            50.00%
#             0.90       0.90   0.9500           1.0556            50.00%
#             0.99       0.99   0.9950           1.0051            50.00%
#     majority 0.99, model 0.995: ratio 1.0051   error reduction 50.0%
#     majority 0.50, model 0.600: ratio 1.2000   error reduction 20.0%
#     against     majority class: ratio 1.2000   error reduction   46.7%
#     against  last year's model: ratio 1.0244   error reduction   11.1%
#     against    random guessing: ratio 1.6800   error reduction   68.0%

A baseline at 0.70 and a model at 0.84 give a ratio of 1.2000 — “twenty percent better.” The same two numbers say the error rate fell from 0.30 to 0.16, a 46.67% reduction in errors. Both are arithmetically correct and they describe the achievement very differently.

The second table shows why the ratio is the weaker of the two. Four settings where the model removes exactly half the remaining errors — the identical achievement each time — give accuracy ratios of 1.5000, 1.2143, 1.0556 and 1.0051. The ratio does not measure the improvement; it measures how much room there was above the majority class.

That compression runs the wrong way for exactly the problems where it matters. On a 99% majority class, halving the errors is a serious result and the ratio calls it 1.005. The third block makes the reversal explicit: a ratio of 1.0051 can represent a 50% error reduction while a ratio of 1.2000 represents 20%.

The fix is to report the relative reduction in the error rate rather than the ratio of accuracies. It is the same two inputs and it does not inherit the floor.

The baseline is the claim

The last block holds one model at 0.84 accuracy and changes only what it is compared against. Against the majority class the error reduction is 46.7%; against last year's model, 11.1%; against random guessing, 68.0%.

The model never changed. A relative number without a named baseline reports a choice of comparison, not a property of the model — and the choice is usually available to whoever is writing the report. This is why a stated improvement should always carry its baseline, and why “beats the baseline” is not a result until the baseline is described.

Two conventions make a comparison defensible. Fix the baseline before the experiment rather than after seeing which comparison flatters the result, and prefer the baseline that would actually be deployed instead — usually the current system, sometimes the majority class, rarely random guessing.

Beyond accuracy

Naming the baseline matters for any relative metric. The compression illustrated above, however, concerns ratios of bounded accuracy with a high majority-class baseline. An error ratio of model error to baseline error is 0.5 whenever errors are halved, whatever the baseline size. Precision and recall have their own denominators and do not inherit the majority-accuracy floor.

Forecasting handles this more carefully than classification does, and it is worth borrowing from. MASE divides test MAE by a naive error measured on training history, with the seasonal lag specified. It is not the percentage improvement against a test-period naive forecast. A relative accuracy figure with a named, pre-registered baseline is doing the same job by discipline instead of by construction.

Finally, relative accuracy says nothing about whether either model is good enough. A model can be 68% better than random and still unusable, and the comparison that decides deployment is against the cost of being wrong, not against another model.

References

• NumPy documentation, array arithmetic. numpy.org