Rogan–Gladen Correction

The Rogan–Gladen estimator corrects a measured rate for the known error rates of the instrument that measured it. Rogan and Gladen proposed it in 1978 to estimate disease prevalence from an imperfect diagnostic test. The same algebra applies when an automated judge, such as an LLM that labels outputs as "unsupported", is used to estimate how often a failure occurs.

The idea

Let \(\pi\) be the true rate of the failure, \(s_e\) the sensitivity of the judge (the share of true failures it flags) and \(s_p\) its specificity (the share of non-failures it passes). The fraction of all cases the judge flags is

\[ q = \pi\, s_e + (1-\pi)(1-s_p). \]

The first term counts true failures that are caught; the second counts good cases that are wrongly flagged. The observed flag rate \(q\) is therefore not \(\pi\). Solving for \(\pi\) gives

\[ \hat\pi = \frac{q + s_p – 1}{s_e + s_p – 1}. \]

Worked example (hypothetical)

A judge has sensitivity 0.90 and specificity 0.95. Plugging in true rates gives these flag rates, and the correction returns the true rate each time:

True rate	Judge flags	Corrected
0.02	0.067	0.020
0.10	0.135	0.100
0.30	0.305	0.300

The table shows why the correction matters most for rare failures: at a true rate of 2% the judge reports 6.7%, more than three times too high, because false flags from the 98% of good cases outnumber the true catches. If a production run shows a flag rate of 0.0775, the corrected rate is (0.0775 + 0.95 − 1) / (0.90 + 0.95 − 1) = 0.0275 / 0.85 = 0.032.

Conditions

• The denominator \(s_e + s_p – 1\) is Youden's J statistic. The correction requires \(J\ne0\). A positive J means failures are more likely to be flagged than non-failures; a negative J reverses that direction but still permits algebraic recovery when the error rates are known. At \(s_e+s_p=1\) the flag rate is the same for every \(\pi\) (for instance 0.6 whether \(\pi\) is 0.1 or 0.5 when \(s_e=0.6\) and \(s_p=0.4\)), so nothing can be recovered. When \(|J|\) is small, the correction amplifies noise.
• The error rates must carry over from the audit set to the data being measured. A judge audited on short, easy outputs may have different rates on long production outputs. The rates should also not depend on the true class mix of the audit sample: sensitivity is estimated from known failures and specificity from known good cases, so the audit may deliberately oversample failures.
• \(s_e\) and \(s_p\) come from finite labeled samples and are uncertain. In a simulation with true rate 0.10, a judge with the rates above, 100 audited failures and 100 audited good cases, and 2,000 production cases, the raw flag rate averaged 0.135 with standard deviation 0.008, while the corrected estimate averaged 0.100 with standard deviation 0.025. The correction removes bias and adds variance, and about 0.1% of the corrected values fell outside [0, 1]. Report an interval, for example by bootstrapping the audit and the production sample together, and clip to [0, 1].

Distinct from nearby ideas

Calibration of predicted probabilities adjusts a score, case by case. Rogan–Gladen adjusts an aggregate rate. It does not tell you which individual flags are wrong. Inter-annotator agreement measures whether humans agree with each other, which is context for the audit labels rather than a correction.