Mean Absolute Error (MAE)
Mean absolute error is the average of the absolute differences between predicted and actual values. In the notation forecasting texts use, with \(e_t = y_t - \hat{y}_t\) the error at time \(t\), it is \(\text{MAE} = \text{mean}(|e_t|)\). The scikit-learn documentation describes it as “a risk metric corresponding to the expected value of the absolute error loss”, the L1 norm of the errors, which says the same thing from the model-training side: it is the average of an L1 penalty. The sign is dropped so that the measure reflects the distance between prediction and actual rather than its direction, which is how the Google crash course describes loss in general.
Two properties follow directly from the definition and explain why the measure is so widely reported. It is in the units of the target, so a MAE of 23 megawatts means what it says. And it is linear in the error, so an error of 40 counts exactly twice an error of 20. The forecasting reference gives those two properties as the reason the measure is so widely used: it is easy to understand and easy to compute.
A worked example
Three hourly electricity demand forecasts, in megawatts, against what happened.
| Hour | Actual | Forecast | Absolute error |
|---|---|---|---|
| 1 | 100 | 90 | 10 |
| 2 | 200 | 220 | 20 |
| 3 | 400 | 360 | 40 |
The MAE is (10 + 20 + 40) / 3 = 23.3 megawatts: on average the forecast is 23.3 megawatts from the actual, in either direction. The same three errors give a root mean squared error of 26.5, larger because the error of 40 is squared before averaging. The gap between the two numbers is itself information: the further RMSE sits above MAE, the more unequal the absolute errors are in size. That is a statement about the sizes of the errors and not about their signs. Errors of +10 and +10 give the same MAE and RMSE as errors of -10 and +10, though the first forecast is biased and the second is not, so the mean error and the distribution of the signed errors have to be examined separately.
What minimizing it means
The fact most often missed is that the error measure decides what the model aims at. The forecasting reference states it directly: “A forecast method that minimises the MAE will lead to forecasts of the median, while minimising the RMSE will lead to forecasts of the mean.” Choosing MAE because it is easier to explain therefore also chooses the median as the target. For a symmetric series the two coincide; for a skewed one, such as demand with occasional spikes, they do not, and a model tuned to MAE will sit below the average on purpose. Which is right depends on the decision downstream. A report of the typical case wants the median. A decision whose cost grows with the square of the miss, in either direction, wants the mean. A stock-level decision, where a unit of shortfall and a unit of excess cost different amounts, wants neither: the target that minimizes expected cost is the quantile fixed by the ratio of the two costs, which is what a quantile loss trains for. The measure only sets the target; a constrained model minimizing MAE lands near the conditional median, not necessarily on it. The entry on forecast error sets out that choice across the whole family of measures, and point forecasts covers what a single number can and cannot represent.
Where it fits, and where it does not
| Question | Answer for MAE |
|---|---|
| Can I compare two models on the same series? | Yes; this is the main use |
| Can I compare across series in different units? | No. Measures based on the raw error “cannot be used to make comparisons between series that involve different units”; use a scaled measure such as MASE or, with care, a percentage such as MAPE |
| How does it treat one very large error? | Linearly. A single miss of 400 adds 400 to the sum, where a squared measure would add 160,000; MAE is less dominated by outliers than MSE or RMSE |
| Does it say whether the model is biased? | No. Over- and under-forecasts add the same amount; check the mean error separately |
| Does it describe how unequal the errors are, or their distribution? | No. “MAE represents the average prediction error”. RMSE beside it shows how unequal the absolute errors are; the distribution of the signed errors has to be looked at directly, and interval forecasts are scored with a quantile loss, which answers a different question |
In code the function is scikit-learn’s mean_absolute_error, and the demand example is three lines:
from sklearn.metrics import mean_absolute_error
mean_absolute_error([100, 200, 400], [90, 220, 360])
# 23.333...The same page notes that when a series mixes very different scales, MAE reflects the error on the largest values and ignores the small ones, which is a case for a percentage or scaled measure rather than a fault in MAE. Report it with the horizon and the data set it was computed on: accuracy on the training set, or one step ahead, is not the accuracy of the forecasts that will be used.
References
- Rob J. Hyndman and George Athanasopoulos, Forecasting: Principles and Practice (3rd ed.), section 5.8 Evaluating point forecast accuracy
- scikit-learn, Metrics and scoring: regression metrics
- Google, Machine Learning Crash Course: Linear regression: Loss
Discover more from Insightful Data Lab
Subscribe to get the latest posts sent to your email.