Root Mean Squared Error (RMSE)

Root mean squared error is the square root of the average squared error: with \(e_t = y_t - \hat{y}_t\), \(\text{RMSE} = \sqrt{\text{mean}(e_t^2)}\). Taking the root puts the number back in the units of the target, which is the reason it exists alongside mean squared error; the Google crash course notes that measures like MAE and RMSE “tend to be more human-interpretable, as they measure error using the same scale as the model’s predicted value.” The squaring inside is what distinguishes it from mean absolute error: “When the difference between the prediction and label is large, squaring makes the loss even larger.”

A worked example, and a common misreading

The mean squared error is (100 + 400 + 100) / 3 = 200, and the RMSE is the square root, about 14.1 units of sales. The misreading to avoid is the forecasts are off by about 14 on average. They are not; the average absolute miss here is (10 + 20 + 10) / 3 = 13.3, the MAE. RMSE is never smaller than MAE, because averaging squares and then taking the root gives more weight to the larger values, and the two are equal only when every error has the same size. The gap between them therefore measures how unequal the absolute errors are, which is what the crash course means when it says RMSE reflects the spread of the errors and “is more skewed by larger errors”. It says nothing about bias: errors of +10 and +10 give the same MAE and RMSE as errors of -10 and +10, though one forecast is biased and the other is not, so the mean error and the distribution of the signed errors have to be examined separately.

What minimizing it means

A model fitted to minimize squared error aims at the conditional mean: the forecasting reference states that “minimising the RMSE will lead to forecasts of the mean”, where minimizing MAE leads to the median. For a series with occasional large values the mean sits above the median, so an RMSE-tuned model forecasts higher, on purpose, than an MAE-tuned one on the same data. That is the right target when the cost of a miss grows with its square, so that large misses are disproportionately expensive and the direction of the miss does not matter. It is the wrong target when the decision needs a typical value, and it is not the target for a stock-level decision either: there a unit of shortfall and a unit of excess cost different amounts, and the forecast that minimizes expected cost is the quantile fixed by the ratio of the two costs, which a quantile loss trains for. The measure sets the target; how close a constrained model gets to the conditional mean is a separate question. The reference is candid about the trade: “Consequently, the RMSE is also widely used, despite being more difficult to interpret.” The entry on forecast error sets the choice of measure in the context of the whole family.

Where it fits, and where it does not

scikit-learn provides root_mean_squared_error directly, so the example is one call:

from sklearn.metrics import root_mean_squared_error
root_mean_squared_error([100, 200, 300], [90, 220, 310])
# 14.142...

As with any accuracy figure, state the data set and horizon: an RMSE computed on training residuals, one step ahead, is not the RMSE of the forecasts that will be acted on.

References

• Rob J. Hyndman and George Athanasopoulos, Forecasting: Principles and Practice (3rd ed.), section 5.8 Evaluating point forecast accuracy
• Google, Machine Learning Crash Course: Linear regression: Loss
• scikit-learn, Metrics and scoring: regression metrics