Bayesian Correction

Prior-odds correction adjusts probabilities when class-conditional feature distributions stay fixed but class proportions change. It applies Bayes’ theorem to accurate source-population probabilities using the source and target priors. A changed class balance alone does not establish these assumptions.

The illustrative function below accepts probabilities and both priors strictly between zero and one. Handle endpoint probabilities separately rather than dividing by zero. It is easiest to state in odds. Multiply the model’s odds by the ratio of the true prior odds to the training prior odds, and convert back:

\[ \text{odds}_{\text{corrected}} = \text{odds}_{\text{model}} \times \frac{\pi_{\text{true}} / (1 - \pi_{\text{true}})}{\pi_{\text{train}} / (1 - \pi_{\text{train}})} \]

It has an exact answer

import numpy as np

def correct(p_model, prior_trained_on, prior_true):
    """Move a probability from the prior the model was fitted under to the real one."""
    odds = p_model / (1 - p_model)
    shifted = odds * (prior_true / (1 - prior_true)) / (prior_trained_on / (1 - prior_trained_on))
    return shifted / (1 + shifted)

print("a classifier trained on a balanced sample reports 0.80 for this email")
for prior in (0.50, 0.30, 0.10, 0.02):
    print(f"  true spam rate {prior:.2f}   corrected probability {correct(0.80, 0.50, prior):.4f}")

print("\nthe same correction applied to a diagnostic test, stated as sensitivity and specificity")
sensitivity, specificity = 0.95, 0.95
for prevalence in (0.50, 0.10, 0.01, 0.001):
    p = sensitivity * prevalence / (sensitivity * prevalence + (1 - specificity) * (1 - prevalence))
    print(f"  prevalence {prevalence:<6.3f}  a positive test means an actual chance of {p:.4f}")
# a classifier trained on a balanced sample reports 0.80 for this email
#   true spam rate 0.50   corrected probability 0.8000
#   true spam rate 0.30   corrected probability 0.6316
#   true spam rate 0.10   corrected probability 0.3077
#   true spam rate 0.02   corrected probability 0.0755
#
# the same correction applied to a diagnostic test, stated as sensitivity and specificity
#   prevalence 0.500   a positive test means an actual chance of 0.9500
#   prevalence 0.100   a positive test means an actual chance of 0.6786
#   prevalence 0.010   a positive test means an actual chance of 0.1610
#   prevalence 0.001   a positive test means an actual chance of 0.0187

A classifier trained on a balanced sample reports 0.80 for an email. Spam is really 10% of the mail. The corrected probability is 0.3077 — odds of 4, multiplied by \(\frac{0.1/0.9}{0.5/0.5} = \frac{1}{9}\), giving odds of \(\frac{4}{9}\).

The transformation is exact when its inputs are the true sampled-population posterior and the correct prior odds under the class-conditional stability assumption. In practice those inputs are estimated and the corrected probability inherits their errors. At a true rate of 0.02, the same assumed source posterior of 0.80 maps to 0.0755; the arithmetic alone does not verify the source model.

The second block is the same arithmetic in the form people meet it most often. A test with 95% sensitivity and 95% specificity sounds decisive. At a prevalence of 1 in 1,000 a positive result means a 1.87% chance of having the condition. Nothing is wrong with the test; the prior is doing the work.

Where the mismatch comes from

• Deliberate resampling. Undersampling the majority class or oversampling the minority changes the training prior on purpose — and the correction above is exactly the prior correction King and Zeng give for case-control samples, which for a logistic model is a shift of the intercept alone.
• Case-control collection. Gathering every case and a sample of controls is standard in epidemiology and fraud, and builds the same mismatch in at the data-collection stage.
• A shifted population. A change from a 3% event rate to 1% can be treated this way only if the class-conditional feature distributions remain stable; prevalence alone does not identify the shift mechanism.

Because the correction multiplies every prediction’s odds by the same constant, it adds a constant to every log-odds — so the ranking is untouched. AUROC, Gini and other rank-based measures of discriminatory power are unchanged by it. It fixes the level, not the order.

What it does not fix

Prior correction assumes the class-conditional feature distributions are unchanged and the source probabilities accurately represent the source population. Correct ordering alone is insufficient: many miscalibrated scores share a ranking. A changed prevalence caused by new class-conditional behavior is not automatically a prior-only shift.

Post-hoc recalibration instead estimates a score-to-outcome mapping on representative held-out data. Platt scaling or isotonic regression can correct some distortions of that relationship; neither can recover information discarded by the score or resolve arbitrary input-dependent changes with a single scalar mapping. Use prior correction when its sampling assumptions hold, then validate any remaining calibration error on the target population.

The label-noise case is a third thing again. When the recorded labels are wrong at known rates, the adjustment runs through a noise transition matrix rather than a prior ratio — a related use of Bayes’ theorem, and a different computation.

Applying it

• Record the training prior. After resampling it is not the prior of the original data, and nothing in a saved model reports it.
• Justify the true prior. The correction is exactly as good as \(\pi_{\text{true}}\), and that number usually comes from a separate estimate with its own uncertainty.
• Correct the intercept, not the outputs, where the model is logistic — one parameter, applied once, rather than a transformation wrapped around every call.
• Check afterward against observed frequencies on data at the natural rate. The correction fixes the intercept; if the slopes were estimated on a small resampled set, some miscalibration survives.

References

• King & Zeng, Logistic Regression in Rare Events Data, Political Analysis 9(2), 2001
• scikit-learn User Guide, Probability calibration