MASE (Mean Absolute Scaled Error)

Mean absolute scaled error divides a forecast’s errors by the size of the errors a naive method makes on the training data, so that the result has no units. The forecasting reference introduces the family as proposed by Hyndman and Koehler in 2006 “as an alternative to using percentage errors when comparing forecast accuracy across series with different units”, and the construction is to scale each error by the mean absolute error of a simple forecasting method on the training data. For a non-seasonal series the scaled error is the forecast error divided by the average absolute one-step change in the training data, \(q_j = e_j \big/ \frac{1}{T-1}\sum_{t=2}^{T} |y_t - y_{t-1}|\), and MASE is the mean of the absolute scaled errors, \(\text{MASE} = \text{mean}(|q_j|)\). For a seasonal series the denominator uses the seasonal naive forecast, the value \(m\) periods earlier.

The reason it is unit-free is stated in one sentence: “Because the numerator and denominator both involve values on the scale of the original data, \(q_{j}\) is independent of the scale of the data.” That is what lets a MASE for a series in megawatts be compared with one in euros, which neither MAE nor RMSE allows, and without the zero and asymmetry problems of MAPE.

What the denominator is, and is not

The interpretation is fixed by the denominator, and the reference is exact about it: “A scaled error is less than one if it arises from a better forecast than the average one-step naïve forecast computed on the training data.” Each qualifier in that sentence is easy to lose. Average: the scale is an average over the whole training period, not the naive error on any particular day. One-step: the naive method is evaluated one step ahead, even when the forecast being scored is twelve steps ahead. Training data: the scale comes from the fitting period, not from the test period the forecast is scored on. A MASE above one therefore means the forecast’s test errors are larger than the naive method’s typical in-sample one-step error; it does not by itself mean the forecast lost to a naive forecast made on the same test period, which is a separate comparison with its own result. The entries on forecast error and the naive baseline forecast cover that comparison and why it is still worth making.

A worked example separates the two periods. A monthly training series of eight values, 100, 120, 130, 150, 170, 160, 180, 200, has one-step absolute changes of 20, 10, 20, 20, 10, 20, 20, whose mean is 17.1: that is the denominator. Three test months then arrive at 210, 230, 220, forecast as 200, 220, 230, so the test MAE is 10 and the MASE is 10 / 17.1 = 0.58. The forecast’s errors are a little over half the size of the naive method’s typical one-step training error. Had the series been monthly with a yearly season, the denominator would be the mean absolute difference from the same month a year earlier, which needs more than twelve months of training data; with eight, there is no pair a year apart, the seasonal scale cannot be computed, and the function below refuses rather than returning a number. The scale must also be positive: a training series that is constant, or that repeats exactly at the chosen lag, gives a zero denominator and no MASE. A reported MASE should therefore say which naive method scaled it and on how much history.

import numpy as np

def mase(y_train, y_test, y_pred, m=1):
    y_train = np.asarray(y_train, dtype=float)
    if len(y_train) <= m:
        raise ValueError("training series must be longer than the lag m")
    scale = np.mean(np.abs(y_train[m:] - y_train[:-m]))   # in-sample naive MAE
    if not scale > 0:
        raise ValueError("naive scale is zero: series repeats exactly at lag m")
    return np.mean(np.abs(np.asarray(y_test) - np.asarray(y_pred))) / scale

y_train = np.array([100, 120, 130, 150, 170, 160, 180, 200])
y_test  = np.array([210, 230, 220])
y_pred  = np.array([200, 220, 230])
mase(y_train, y_test, y_pred)   # 0.583...

Where it fits

References

• Rob J. Hyndman and George Athanasopoulos, Forecasting: Principles and Practice (3rd ed.), section 5.8 Evaluating point forecast accuracy