Forecast Error

1. Definition

• A forecast error is the difference between an observed value and the value forecast before that observation became available.

$e_{T+h} = y_{T+h} – \hat{y}_{T+h\mid T}$

Where:

• $y_{T+h}$ = actual value observed at forecast horizon $h$.
• $\hat{y}_{T+h\mid T}$ = forecast for time $T+h$ using information available through time $T$.
• $e_{T+h}$ = forecast error.

The sign convention can vary across organizations. This article uses actual minus forecast, so positive errors indicate underforecasting.

Forecast Errors vs. Residuals

• Forecast errors are calculated from genuine forecasts, typically on validation or test observations not used to fit the model.
• Residuals are calculated from fitted or one-step-ahead values within the training data.
• Residual diagnostics help assess model fit, but small training residuals do not guarantee accurate forecasts on new data.

---

2. Interpretation

• If $e_t > 0$, the forecast underestimated the actual value.
• If $e_t < 0$, the forecast overestimated the actual value.
• If $e_t = 0$, the forecast exactly matched the observed value.
• A large positive or negative error indicates a large miss, but its practical importance depends on the scale and business context.

Signed errors can cancel one another when averaged. For this reason, forecast evaluation usually includes absolute, squared, percentage, or scaled error metrics.

---

3. Properties of Useful Forecast Errors and Residuals

• Low bias: the mean error should be close to zero over a relevant evaluation period.
• Little predictable structure: residuals should not contain substantial autocorrelation or patterns that the model could have used.
• Appropriate uncertainty: prediction intervals should reflect changes in forecast uncertainty across horizons and conditions.
• Stability: error behavior should be monitored across time, products, locations, horizons, and other important segments.

Constant variance is useful for some statistical models and simplifies interval estimation, but it is not a universal requirement for accurate forecasting. Changing variance may be an inherent feature of the data and can be modeled explicitly.

Bias, autocorrelation, trends, seasonality, or changing variance can indicate missing structure, data drift, model limitations, or changes in the underlying process. They do not automatically prove that the model is misspecified.

---

4. Example

Suppose actual and forecast sales are measured in units:

Time	Actual ($y_t$)	Forecast ($\hat{y}_t$)	Error ($e_t$)
1	100	95	+5 (underforecast)
2	120	130	−10 (overforecast)
3	90	88	+2 (underforecast)

The errors are $[5, -10, 2]$.

• $ME = (5 – 10 + 2)/3 = -1$ unit.
• $MAE = (5 + 10 + 2)/3 \approx 5.67$ units.
• $MSE = (5^2 + (-10)^2 + 2^2)/3 = 43$ squared units.
• $RMSE = \sqrt{43} \approx 6.56$ units.

The mean error is close to zero because positive and negative errors partially cancel, while MAE and RMSE reveal the typical magnitude of the misses.

---

5. Forecast Error Metrics

Raw errors can be positive or negative, so forecast performance is summarized using several complementary metrics:

• Mean Error (ME): measures average signed error and is commonly used to assess bias.
  • $ME = \frac{1}{n}\sum_{t=1}^{n} e_t$
• Mean Absolute Error (MAE): measures the average absolute error in the original units.
  • $MAE = \frac{1}{n}\sum_{t=1}^{n}|e_t|$
• Mean Squared Error (MSE): averages squared errors and gives greater weight to large misses.
  • $MSE = \frac{1}{n}\sum_{t=1}^{n}e_t^2$
• Root Mean Squared Error (RMSE): is the square root of MSE and returns the metric to the original units.
  • $RMSE = \sqrt{\frac{1}{n}\sum_{t=1}^{n}e_t^2}$
• Mean Absolute Percentage Error (MAPE): expresses absolute errors as percentages of actual values.
  • $MAPE = \frac{100}{n}\sum_{t=1}^{n}\left|\frac{e_t}{y_t}\right|$
  • MAPE is undefined when any actual value is zero and can become unstable when actual values are close to zero.
• Mean Absolute Scaled Error (MASE): scales MAE using the in-sample error of a naïve benchmark, making results more comparable across series with different units.

No metric is universally best. The appropriate choice depends on the decision problem, error costs, data scale, presence of zeros, forecast horizon, and comparison objective.

---

6. In Forecasting Competitions

Forecasting competitions use predefined metrics to compare methods consistently, but the metrics differ across competitions.

• M4 Competition: point forecasts were evaluated using sMAPE, MASE, and the Overall Weighted Average (OWA).
• M5 Accuracy Competition: used the Weighted Root Mean Squared Scaled Error (WRMSSE).
• A better result means a lower value of the competition’s selected loss or error metric, not necessarily a raw signed error closer to zero.

---

Summary:
Forecast error is the difference between an observed value and a forecast produced before that value was known. Under the actual-minus-forecast convention, positive errors indicate underforecasting and negative errors indicate overforecasting.

• Forecast errors should be distinguished from training residuals.
• Useful forecasts generally have low bias and leave little predictable structure in their errors.
• Metrics such as MAE, RMSE, MAPE, and MASE summarize different aspects of forecast performance.
• Metric selection should reflect the data, decision costs, and evaluation objective.