Diagnostics After Fitting a Time Series Model

1) Why diagnostics matter

After you fit a time series model to data, the central question becomes:

Did the model actually capture the predictable structure in the data, or is there still systematic pattern left over?

The model’s “leftovers” are the residuals (prediction errors). If the model is appropriate, those residuals should look like:

• white noise (no time dependence left),
• with constant variance, and
• (under the Gaussian assumption) approximately normal.

So diagnostics focus on two main checks:

1. Distribution check: Are residuals approximately normal?
2. Autocorrelation check: Are residuals approximately uncorrelated over time?

If residuals fail these checks, it suggests the fitted model is missing something (wrong order, missing seasonal structure, missing differencing, nonconstant variance, non-Gaussian shocks, outliers, structural change, etc.).

---

2) Standardized residuals: definition and meaning

2.1 One-step-ahead prediction and prediction error

Consider a mean-zero stochastic process $(X_t)$. For each time $n+1$, define the best linear prediction of $X_{n+1}$​ using all available past observations $X_1,\dots,X_n$​:

$$\hat{X}_{n+1} = P(X_{n+1}\mid X_1,\dots,X_n)$$

This is a linear combination of past values:

$$\hat{X}_{n+1} = \phi_{n,1}X_n + \phi_{n,2}X_{n-1} + \cdots + \phi_{n,n}X_1$$

The one-step-ahead prediction error is:

$$X_{n+1}-\hat{X}_{n+1}$$

and its variance is defined as:

$$v_n = E\left[(X_{n+1}-\hat{X}_{n+1})^2\right]$$

2.2 Standardizing the errors

Raw prediction errors can have different variances depending on time index (especially early in a series, or depending on the model’s internal recursion). To make them comparable, divide each error by its standard deviation:

$$e_n = \frac{X_n – \hat{X}_n}{\sqrt{v_{n-1}}},\quad n=1,\dots,N$$

These $e_n$​ are called standardized residuals.

2.3 Why standardized residuals should look like variance-1 white noise

Three key properties follow directly from this construction:

1. Mean is 0
   The predictor $\hat{X}_n$​ has the same mean as $X_n$​, so $E(X_n-\hat{X}_n)=0$. Dividing by $\sqrt{v_{n-1}}$​​ keeps the mean 0.
2. Variance is 1
   By definition, $v_{n-1} = E[(X_n-\hat{X}_n)^2]$, so dividing by $\sqrt{v_{n-1}}$​​ forces: $$\operatorname{Var}(e_n)=1$$
3. No correlation across time
   A standard result for optimal one-step prediction errors in this setting is that they are pairwise uncorrelated. Scaling by constants does not change correlation, so: $$\operatorname{corr}(e_n,e_m)=0 \quad \text{for }n\neq m$$

Putting these together: $(e_n)$ satisfies the definition of white noise with variance 1.

2.4 What you actually have in data

You do not observe $X_t$​; you observe one realization $x_1,\dots,x_N$. If the fitted model truly generated the data, then the computed standardized residuals:

$$e_n(\omega)=\frac{x_n-\hat{x}_n}{\sqrt{v_{n-1}}}$$

should behave like one realized path of variance-1 white noise—meaning:

• the residual ACF should look like noise,
• the residual distribution should be close to normal (if Gaussian noise is assumed).

If not, that contradicts the “this model generated the data” idea and signals model inadequacy.

---

3) Residuals vs standardized residuals (the scaling issue)

In many software outputs you may see both:

• Unstandardized residuals: $x_n-\hat{x}_n$​
• Standardized residuals: $(x_n-\hat{x}_n)/\sqrt{v_{n-1}}$​​

A common simplification for fitted ARMA models is that the prediction error variance is proportional to the estimated shock variance $\hat{\sigma}^2$. In that case, dividing by $\sqrt{\hat{\sigma}^2}$​ produces a version that differs only by a constant scaling from the fully standardized form. Because of that, plots of residuals and standardized residuals usually look the same up to vertical rescaling.

The important point is interpretability:

• Standardized residuals are on a common variance-1 scale.
• That makes normality checks and “is this unusually large?” reasoning more meaningful.

---

4) What diagnostic plots are trying to show (high-level interpretation)

A typical diagnostic output after fitting a model includes:

1. Standardized residual time plot
   Quick visual screening for:
   • remaining trend or seasonality (systematic patterns),
   • changing variance over time (volatility clustering),
   • outliers or bursts.
   If the model is adequate, this plot should look like random fluctuations around 0 with no obvious structure.
2. Residual ACF plot
   If the model captured the dependence structure, residual autocorrelations should be small and mostly within approximate confidence bands. One isolated spike can happen by chance, but many spikes or systematic tailing indicates remaining structure.
3. Normal Q–Q plot
   Checks whether residuals are approximately normal.
4. A plot or table of Ljung–Box p-values across lags
   Formalizes the residual autocorrelation check across multiple lags simultaneously.

---

5) Normal Q–Q plot: what it checks and how to read it

5.1 Purpose

A Q–Q plot compares:

• the ordered residuals (your data),
  against
• the ordered quantiles of a target distribution (here: standard normal).

If residuals are normally distributed, the points fall approximately on a straight line.

This is not a formal proof; it is a visual diagnostic that is very good at showing:

• heavy tails,
• skewness,
• outliers.

5.2 What each point represents (point-by-point meaning)

Let standardized residuals be $e_1,\dots,e_N$​. Sort them:

$$e_{(1)} \le e_{(2)} \le \cdots \le e_{(N)}$$

Now generate $N$ target probabilities designed to spread evenly across $(0,1)$, for example:

$$q_k = \frac{2k-1}{2N}, \quad k=1,\dots,N$$

Compute the corresponding normal quantiles:

$$z_k = \Phi^{-1}(q_k)$$

where $\Phi^{-1}$ is the inverse CDF of $N(0,1)$.

Then the Q–Q plot points are:

$$(z_k,\ e_{(k)})$$

So the $k$-th smallest residual is paired with the $k$-th normal quantile.

5.3 How to interpret shapes

• Straight line: residual distribution is close to normal.
• Curving outward at both tails: residuals have heavier tails than normal (more extreme values).
• One tail deviates more: skewness.
• A few points far from the line: outliers.

A small tail deviation is common in real data and not automatically disqualifying; the question is whether the deviations are large enough to matter for your modeling goals.

---

6) Ljung–Box test: a formal check for residual autocorrelation

6.1 Why the residual ACF plot is not enough

ACF plots usually show approximate 95% confidence bands for each lag separately. But you can have a situation where:

• every lag is “barely” inside the band,
• yet collectively the residuals show meaningful correlation.

The Ljung–Box statistic aggregates information across many lags at once.

6.2 The Ljung–Box Q-statistic

Let $\rho_e(h)$ be the sample autocorrelation of the standardized residuals at lag $h$. For a chosen maximum lag $H$, define:

$$Q(H)= n(n+2)\sum_{h=1}^{H}\frac{\rho_e(h)^2}{n-h}$$

where $n$ is the length of the residual series.

This statistic becomes large when the residual autocorrelations are collectively large.

6.3 Hypothesis test and degrees of freedom correction

When you fit an ARMA(p,q) model, you have already used the data to estimate $p+q$ parameters (plus possibly a mean). This affects how you compare the test statistic to a reference distribution.

The standard approach is:

• Null hypothesis: residuals are uncorrelated up to lag $H$.
• Compare $Q(H)$ to a chi-square distribution with: $$\text{df} = H – p – q$$

A small p-value indicates you reject the null and conclude there is remaining autocorrelation structure in residuals—meaning the model did not fully capture time dependence.

6.4 Interpreting a p-value plot over many lags

If you compute the Ljung–Box p-value for each $H$ in a range (e.g., $H=3,\dots,20$) and plot them:

• If p-values are mostly above a threshold like 0.05, residuals look consistent with white noise across those lag ranges.
• If many p-values drop below 0.05, that is evidence of leftover dependence.

---

7) What it means if diagnostics “fail”

If diagnostics show problems, typical implications are:

• Residual ACF significant: model order is too low, or wrong AR/MA balance, or missing seasonal terms, or missing differencing, or structural break.
• Residual variance changes over time: consider transformations (log), variance models, or identify regime changes/outliers.
• Q–Q plot shows heavy tails/outliers: Gaussian assumption may be imperfect; inference based on normality may be less reliable; robust alternatives may be needed.

Diagnostics are not just about “passing tests.” They guide model refinement.

---

8) Practical diagnostic workflow (what you should check every time)

After fitting an ARMA/SARIMA-type model:

1. Plot standardized residuals over time (look for obvious structure).
2. Plot residual ACF (look for remaining autocorrelation).
3. Produce a normal Q–Q plot (check normality).
4. Run Ljung–Box over a range of $H$ values (check joint correlation).
5. If anything looks problematic, revise model structure and repeat.