Time Series Forecasting

Time series forecasting is using a series’ own history, and sometimes other information, to say what its future values will be. What separates it from ordinary regression is that the observations are ordered, each is correlated with the ones before it, and the structure that makes prediction possible — trend, seasonality — is also what makes the usual independence assumptions false. The series itself and its components are covered under time series.

Benchmarks come first

The most useful discipline in forecasting is also the cheapest: before fitting anything, produce the forecasts a method with no model would give, and require everything afterward to beat them. Hyndman and Athanasopoulos set out four such methods and are explicit about their role — “sometimes one of these simple methods will be the best forecasting method available; but in many cases, these methods will serve as benchmarks rather than the method of choice”, and a new approach should be compared against them because “if not, the new method is not worth considering”.

The series below is synthetic: eight years of monthly data with a rising trend, a fixed twelve-month pattern and noise. The last two years are held out.

import numpy as np

def monthly_series(years=8, seed=12):
    """Monthly data with a rising trend, a fixed 12-month seasonal pattern, and noise."""
    rng = np.random.default_rng(seed)
    t = np.arange(12 * years)
    season = 20 * np.sin(2 * np.pi * t / 12)
    return 100 + 0.5 * t + season + rng.normal(0, 5, t.size)

def mean_absolute_error(actual, predicted):
    return float(np.abs(np.asarray(actual) - np.asarray(predicted)).mean())

y = monthly_series()
train, test = y[:-24], y[-24:]
horizon, m = len(test), 12

naive = np.repeat(train[-1], horizon)
mean_method = np.repeat(train.mean(), horizon)
# Seasonal naive: each forecast is the last observation from the same season.
seasonal_naive = np.array([train[-m + (h % m)] for h in range(horizon)])
drift_slope = (train[-1] - train[0]) / (len(train) - 1)
drift = train[-1] + drift_slope * np.arange(1, horizon + 1)

for name, forecast in (("naive", naive), ("mean", mean_method),
                       ("seasonal", seasonal_naive), ("drift", drift)):
    print(f"{name:9} {mean_absolute_error(test, forecast):6.2f}")
# naive 17.51
# mean 24.08
# seasonal 11.78
# drift 14.69

The ordering is informative. Seasonal naïve wins because the series is strongly seasonal and it is the only benchmark that knows this, and drift comes second because it is the only one that knows about the trend. Neither knows about both, which is exactly the gap a real model should close — and a model that lands above 11.78 on this data has not earned the run. Naïve baseline forecast covers the benchmarks and the traps in comparing against them.

Residuals are not forecast errors

This distinction quietly invalidates a great deal of reported accuracy. “Forecast errors are calculated on the test set while residuals are calculated on the training set”, and residuals involve one-step-ahead predictions while forecast errors can be many steps ahead. A model evaluated on its residuals is being asked a much easier question than the one it will face, twice over: it has seen the data, and it only has to predict one step.

So the split is not optional, and it cannot be random. Holding out a random 20 percent of a series leaves gaps that neighboring observations fill in, which measures interpolation rather than forecasting. The test set is the end of the series. The usual sizing is that “the size of the test set is typically about 20% of the total sample”, and it should be at least as long as the largest horizon you intend to forecast — otherwise the evaluation never tests the thing being asked for.

For a more thorough version, time series cross-validation rolls the split forward: fit on everything up to a point, forecast the next \(h\) steps, move the point along, repeat. It gives many more error observations than a single split, at the cost of refitting many times.

Which error measure

Mean absolute error and root mean squared error are in the units of the series, which makes them easy to interpret on one series and useless for comparing across series measured in different units. That is the reason scaled and percentage-based measures exist, and the reason a forecasting system reporting one aggregate number across a catalog of products needs a scale-free measure to do it honestly.

The choice between absolute and squared error is a choice about what a large miss costs. Squared error penalizes a few large misses heavily and will prefer a model that is consistently a little wrong; absolute error is indifferent between one miss of ten and ten misses of one. Picking the one that matches the decision, rather than the one the library defaults to, is worth the two minutes it takes.

Point forecasts and intervals

A point forecast is a single number and is almost never the whole answer. An inventory decision needs to know how bad the bad case is; a capacity decision needs the upper tail. Prediction intervals supply that, and they behave in a specific way: “a common feature of prediction intervals is that they usually increase in length as the forecast horizon increases. The further ahead we forecast, the more uncertainty is associated with the forecast, and thus the wider the prediction intervals.”

Intervals computed from a fitted model take that model as given, so they reflect the uncertainty the model knows about and not the uncertainty of having chosen the wrong model or of the pattern changing. That is one reason forecast intervals tend to be optimistic in practice, and a reason to check their realized coverage on held-out data rather than trusting the nominal level.

The shape of the work

• Look at the series. Plot it, plot it by season, and plot the autocorrelation function. Most decisions about method are visible here — see temporal autocorrelation.
• Fix the problems that are not modeling problems. Missing intervals, changed collection frequency, a variance that grows with the level. A logarithm stabilizes the last of these and is usually easier than a model that accommodates it.
• Establish the benchmarks. Before any model.
• Fit, on the training portion only.
• Evaluate on the held-out end of the series, at the horizon the decision actually uses, and check the residual autocorrelation — structure left in the residuals is signal the model did not take.

The method families differ in what they assume rather than in their sophistication. Exponential smoothing and the ARIMA family encode structure explicitly and produce intervals from the model. Regression and tree-based methods see no time structure unless lags and calendar features are engineered for them, and repay that work when external drivers matter. Sequence models learn dependence from data and need a lot of it, which is why they tend to earn their place across many related series rather than on one. Curve-fitting approaches such as Prophet support automated forecasts and analyst adjustment while aiming for accurate predictions. Compare accuracy and runtime on the series that matters: Prophet was less accurate in the two FPP3 examples, and the electricity-demand example was faster to estimate than the DHR comparison.

One final caution about what a forecast is for. A forecast says what would happen if the conditions that generated the history continue. Comparing a forecast with what actually happened, and attributing the gap to something that changed, is a causal claim that rests on the forecast having been right — a far stronger requirement than it being useful.

References

• Rob J. Hyndman and George Athanasopoulos, Forecasting: Principles and Practice (3rd edition), section 5.2 Some simple forecasting methods, OTexts
• Rob J. Hyndman and George Athanasopoulos, Forecasting: Principles and Practice (3rd edition), section 5.8 Evaluating point forecast accuracy, OTexts
• Rob J. Hyndman and George Athanasopoulos, Forecasting: Principles and Practice (3rd edition), section 5.5 Prediction intervals, OTexts