Mean Absolute Percentage Error (MAPE)

Mean absolute percentage error expresses each error as a share of the actual value and averages the absolute shares. The forecasting reference defines the percentage error as \(p_t = 100 e_t / y_t\), with \(e_t = y_t - \hat{y}_t\), and MAPE as \(\text{mean}(|p_t|)\). The appeal is stated in the same place: “Percentage errors have the advantage of being unit-free, and so are frequently used to compare forecast performances between data sets.” The scikit-learn documentation puts the property as invariance: the measure “is for example not changed by a global scaling of the target variable.”

Three product-demand forecasts show the reading. Actuals of 100, 200, and 400 against forecasts of 90, 220, and 360 give absolute errors of 10, 20, and 40, which are 10 percent of each actual, so the MAPE is 10 percent: on average the forecast is ten percent from the actual. The same forecasts have a mean absolute error of 23.3 units, a number that could not be compared with a MAE from a series measured in a different unit, whereas the 10 percent could.

What the percentage costs

The convenience has a price, and the reference lists most of it. The measure is “infinite or undefined if \(y_{t}=0\) for any \(t\) in the period of interest”, and has “extreme values if any \(y_{t}\) is close to zero”: one week of zero sales makes the summary meaningless, and a handful of near-zero actuals can dominate the average. Percentages “assume the unit of measurement has a meaningful zero”, so a percentage error on a temperature forecast means nothing on the Celsius or Fahrenheit scale, whose zero points are conventions. And the measure is asymmetric in the sense the reference gives: percentage errors “put a heavier penalty on negative errors than on positive errors”. For a single fixed actual the score is symmetric, since forecasts of 50 and 150 against an actual of 100 are both 50 percent errors. The asymmetry arises across the actuals a series takes, because each term is weighted by one over its actual: the same absolute miss of 50 is a 50 percent error when the actual is 100 and a 33 percent error when the actual is 150, so the periods with small actuals carry the heavier weight. A high forecast in a period whose actual turns out small produces a large percentage, while a low forecast in a period whose actual turns out large produces a small one, and over a series whose actuals vary a model can therefore improve its expected MAPE by shading its forecasts downward. How strong that incentive is depends on how much the actuals vary relative to the errors. One further cost is not in the reference’s list but follows from the definition: a percentage error weights a small item and a large one equally, so a 10 percent miss on ten units and on ten thousand count the same, which is rarely how the cost falls.

The usual repairs are weaker than they look. Adding a small constant to the denominator defines a different measure whose value depends on the constant. The signed-denominator version of sMAPE used in the reference averages \(200|y_t - \hat{y}_t|/(y_t + \hat{y}_t)\). Its terms can be negative when the denominator is negative and undefined when it is zero. Another version uses \(|y_t| + |\hat{y}_t|\) in the denominator: its defined terms are non-negative, but a zero actual and zero forecast still require an explicit convention. The reference warns about the near-zero denominator in its version and cites Hyndman and Koehler’s recommendation against using it. Reporting sMAPE therefore requires stating which formula and zero handling are used. A weighted variant that divides the sum of absolute errors by the sum of actuals, often called WAPE, avoids the per-item division and weights large items more, which suits inventory-style questions; and a scaled measure such as MASE compares across series without any percentage at all. Which to choose is set out under forecast error.

Implementation detail that changes the number

scikit-learn’s mean_absolute_percentage_error returns a fraction, not a percentage; the documentation states that its formula “does not represent the common” percentage definition, and multiplying by 100 recovers the percentage. Each absolute error is divided by \(\max(\epsilon, |y_i|)\), the larger of the absolute actual and a tiny positive constant chosen “to avoid undefined results when y is zero”. The consequence for zeros is specific: an actual of zero with a non-zero prediction contributes a very large finite term rather than raising an error, and an actual of zero predicted as zero contributes nothing, so a series with zeros returns a number whose size depends on what was predicted at those points. The demand example returns 0.1, which is the 10 percent above.

from sklearn.metrics import mean_absolute_percentage_error
mean_absolute_percentage_error([100, 200, 400], [90, 220, 360])
# 0.1  (a fraction: 10 percent)

References

• Rob J. Hyndman and George Athanasopoulos, Forecasting: Principles and Practice (3rd ed.), section 5.8 Evaluating point forecast accuracy
• scikit-learn, Metrics and scoring: regression metrics