Continuous Ranked Probability Score (CRPS)
CRPS reads like a weighted absolute error computed from the entire distribution, so it rewards being both accurate and appropriately confident.
-
-
Pinball Loss (a.k.a. Quantile Loss)
At p = 0.5 it is just absolute error. Away from the median it weights misses unevenly, which is what makes a quantile the thing being estimated.