Strictly Proper Scoring Rules
The property that makes a probability metric ungameable: honest reporting is uniquely optimal. Shown by calculation, including a rule that lacks it.
-
-
Binary Cross-Entropy (BCE)
On events that happen 30% of the time, reporting 0.30 gave the lowest average loss at 0.610682. Reporting 0.20 or 0.40 both cost more.
-
Log Loss
The negative log-likelihood, used as a metric. Strictly proper, unbounded, and sensitive enough to a single confident error to need watching.
-
Probability Calibration
Ranking well and reporting honest probabilities are different skills.