Understanding ACFs via Difference Equations for AR(p) and ARMA(p, q)

1) Big picture: what we are trying to compute

• A time series is $(X_t)$.
• The ACF at lag $h$, written $\rho_X(h)$, measures how strongly $X_t$ is correlated with $X_{t-h}$.
• We always have $\rho_X(0)=1$ and (for stationary series) $\rho_X(-h)=\rho_X(h)$.

The strategy here is:

1. Start from the model equation (AR or ARMA).
2. Multiply both sides by $X_{t-h}$​, take expectations.
3. Use properties of white noise and causality to simplify.
4. Obtain a recurrence relation for $\gamma_X(h)$ (autocovariance) or directly for $\rho_X(h)$ (autocorrelation).
5. Solve the recurrence using a small set of “initial values.”

---

2) ACF of a causal AR(p): the recurrence and why you need $p$ constants

The AR(p) model

$$X_t – \phi_1 X_{t-1} – \cdots – \phi_p X_{t-p} = W_t,$$

where $(W_t)$ is white noise with variance $\sigma_W^2$​.

Key step: multiply by $X_{t-h}$​ and take expectations

For causal AR(p), when $h \ge 1$, the term $E(W_t X_{t-h}) = 0$ because $X_{t-h}$​ depends only on past shocks and is uncorrelated with the “new shock” $W_t$​.

That produces the recurrence (for autocovariances):$$\gamma_X(h)=\phi_1\gamma_X(h-1)+\cdots+\phi_p\gamma_X(h-p),\quad h\ge 1.$$

Divide by $\gamma_X(0)$ to get the ACF recurrence:$$\rho_X(h)=\phi_1\rho_X(h-1)+\cdots+\phi_p\rho_X(h-p),\quad h\ge 1.$$

Why you need $p$ “starting values”

This is a p-th order recurrence. To generate $\rho(1),\rho(2),\ldots$ you need:

• $\rho(0)=1$ plus
• $\rho(1),\ldots,\rho(p-1)$

Those $\rho(1),\ldots,\rho(p-1)$ are found by plugging $h=1,2,\ldots,p-1$ into the recurrence and using symmetry $\rho(-k)=\rho(k)$. That yields a linear system of $p-1$ equations in $p-1$ unknowns.

What the closed-form solution looks like

Let the AR polynomial be$$\phi(z)=1-\phi_1 z-\cdots-\phi_p z^p.$$

If its roots are $r_1,\ldots,r_p$​, then (in the distinct-root case) solutions look like sums of terms:$$\rho(h)=c_1 r_1^{-h}+ \cdots + c_p r_p^{-h}.$$

If a root is repeated, you get polynomial factors in $h$. For example, if a root $r$ has multiplicity $k$, it contributes:$$(\text{polynomial of degree }k-1 \text{ in } h)\cdot r^{-h}.$$

If there are complex conjugate roots, the ACF becomes a damped oscillation (a decaying cosine wave).

Causality condition and decay

Causality for AR(p) corresponds to all roots satisfying $|r_j|>1$. Then $|r_j^{-h}|\to 0$, so the ACF decays to 0 as $h\to\infty$.

---

3) Example: AR(3) and how initial conditions are computed

For an AR(3),$$X_t-\phi_1X_{t-1}-\phi_2X_{t-2}-\phi_3X_{t-3}=W_t,$$

the ACF recurrence is:$$\rho(h)=\phi_1\rho(h-1)+\phi_2\rho(h-2)+\phi_3\rho(h-3),\quad h\ge 1.$$

To determine $\rho(1)$ and $\rho(2)$, plug in $h=1$ and $h=2$, then use $\rho(-1)=\rho(1)$, $\rho(-2)=\rho(2)$, and $\rho(0)=1$. This produces a 2×2 linear system in $\rho(1),\rho(2)$. Once those are known, the recurrence generates all larger lags.

Then, if you want an explicit closed-form formula, you:

1. factor $\phi(z)$,
2. write the general form (including $h$-polynomials for repeated roots),
3. plug in $\rho(0),\rho(1),\rho(2)$ to solve for the constants.

---

4) How to compute the variance $\gamma_X(0)$ once you know the ACF

Working with $\rho(h)$ gives correlations, but not the variance $\gamma(0)=\mathrm{Var}(X_t)$.

For a causal AR(p), one can derive:$$\gamma_X(0)\left(1-\sum_{h=1}^{p}\phi_h\,\rho_X(h)\right)=\sigma_W^2,$$

so$$\boxed{\gamma_X(0)=\frac{\sigma_W^2}{1-\sum_{h=1}^{p}\phi_h\,\rho_X(h)}}.$$

Interpretation:

• The variance of $X_t$​ is the white-noise variance “amplified” by feedback from past values, captured by the $\phi_h$​ and the ACF values.

---

5) What changes for ARMA(p, q): why “small lags” become special

ARMA(p, q) model

$$\phi(B)X_t=\theta(B)W_t,$$

where

• $\phi(B)=1-\phi_1B-\cdots-\phi_pB^p$,
• $\theta(B)=1+\theta_1B+\cdots+\theta_qB^q$.

A causal ARMA process can be written as a linear process:$$X_t=\sum_{j=0}^{\infty}\psi_j W_{t-j},\quad \psi_0=1.$$

The “minor complication”

When you repeat the earlier step (multiply by $X_{t-h}$, take expectations), the right-hand side now contains terms like $E(W_t X_{t-h})$ and $E(W_{t-1}X_{t-h})$, etc.

• For large lags $h$, these expectations become 0.
• But for small lags (roughly $h \le q$ and nearby), they are not necessarily 0.

Result:

• For sufficiently large $h$, the ACF (or autocovariance) satisfies the same AR-type recurrence

$$\gamma(h)-\phi_1\gamma(h-1)-\cdots-\phi_p\gamma(h-p)=0.$$

• But to “start” the recurrence, you must solve a finite system for the first few $\gamma(0),\ldots,\gamma(r-1)$, where

$$r=\max\{p,\;q+1\}.$$

Those equations depend on the first few $\psi_j$​ coefficients.

Practical point: getting the $\psi_j$

For ARMA(1,1), $\psi_0,\psi_1,\ldots$ can be derived by hand. In general ARMA(p,q), doing this manually is tedious, so software computes $\psi_j$​ numerically (e.g., converting ARMA to an MA($\infty$) representation).

Conceptually:

• $\psi_j$ tells you how a shock $W_{t-j}$​ flows into $X_t$​.
• You only need the first several $\psi_j$​ to set up the initial autocovariance equations.

---

6) Recognizing model classes from ACF shape

MA(q): sharp cutoff

For MA(q),$$X_t=W_t+\theta_1W_{t-1}+\cdots+\theta_qW_{t-q},$$

the ACF is exactly zero beyond lag $q$:$$\rho(h)=0\quad\text{for } h>q.$$

So MA(q) is often visually identifiable because the ACF cuts off.

AR(p): no cutoff, but decay (possibly oscillatory)

For causal AR(p), the ACF does not cut off. It decays toward 0, often as a mixture of exponentials and possibly damped oscillations (if complex roots exist).

ARMA: also no cutoff, and can resemble AR

Causal ARMA typically also has an ACF that tails off (no sharp cutoff). This makes it harder to distinguish AR vs ARMA using ACF alone—both can show gradual decay.

That is why another function is introduced in the next step of the typical toolkit: PACF (partial autocorrelation), which is especially helpful for identifying AR order.

---

7) Summary

Difference equations provide a clean way to compute ACFs for AR and ARMA models: for AR(p), the ACF satisfies a p-th order linear recurrence whose solution is built from the roots of the AR polynomial (with polynomial-in-$h$ factors when roots repeat and damped oscillations when roots are complex). The unknown constants are pinned down by the initial values $\rho(0),\ldots,\rho(p-1)$, found via a linear system. For ARMA(p,q), the same AR-type recurrence holds after a certain lag, but the first few autocovariances must be solved from additional equations that depend on the MA side; this requires a few $\psi_j$​ coefficients from the MA($\infty$) representation. In plots, MA(q) is notable for a hard cutoff in the ACF, while AR and ARMA generally show tailing decay, making them harder to separate using ACF alone.