ARIMA stands for AutoRegressive Integrated Moving Average. It’s a classic, general‑purpose model for univariate time‑series forecasting that captures three ideas: autocorrelation (AR), differencing to remove trends (I), and short‑memory shock smoothing (MA).

1) Intuition first

  • AR (AutoRegressive): Today’s value is partly explained by past values.
    “Sales today look like a weighted average of sales 1–p days ago.”
  • I (Integrated): If the series trends or wanders, first difference it to make it stable (“stationary”).
    “Model the changes rather than the raw level.”
  • MA (Moving Average): Today’s value includes a component from past shocks/errors.
    “One‑off surprises ripple into the next few periods.”

2) The parts and the notation

  • Model name: ARIMA(p, d, q)
    • $p$: AR order (lags of $x_t$​)
    • $d$: differencing order (how many times we difference to stationarize)
    • $q$: MA order (lags of the error term)

Using the backshift operator $B x_t = x_{t-1}$​ and difference operator $(1-B)$,
the model can be written as:

$\phi(B)\,(1-B)^d x_t = c + \theta(B)\,\varepsilon_t,$

where $\varepsilon_t$​ is white noise,
$\phi(B) = 1 – \phi_1 B – \cdots – \phi_p B^p$,
$\theta(B) = 1 + \theta_1 B + \cdots + \theta_q B^q$.

Stationarity (AR side) and invertibility (MA side) require the roots of $\phi(B)$ and $\theta(B)$ to lie outside the unit circle.

3) Why differencing (“I”) matters

  • Many economic and operational series are non‑stationary (mean or variance changes over time).
  • Differencing $d$ times applies $(1-B)^d$ to $x_t$​.
    • $d=1$ models first differences $\Delta x_t = x_t – x_{t-1}$​.
    • $d=2$ models differences of differences, etc.
  • Goal: a series whose mean/variance are roughly stable and whose autocorrelations die out.

Tip: Use minimal differencing that achieves stationarity. Over‑differencing inflates variance and can create spurious dynamics.

4) Model identification with ACF/PACF (Box–Jenkins workflow)

  1. Make the series stationary (transform + difference):
    • Stabilize variance (e.g., log) if needed.
    • Difference until the Augmented Dickey–Fuller suggests stationarity (or visually: mean flat, ACF tails decay).
  2. Inspect ACF/PACF of the stationary series:
    • AR(p): PACF cuts off after lag $p$; ACF decays gradually.
    • MA(q): ACF cuts off after lag $q$; PACF decays gradually.
    • ARMA(p,q): both ACF and PACF decay (no sharp cut‑off).
  3. Try a few plausible $(p,d,q)$ and pick with AIC/AICc/BIC.

5) Estimation and diagnostics

  • Parameters are typically estimated via maximum likelihood (often with Kalman filter under the hood) or conditional least squares.
  • Residual diagnostics are essential:
    • Residuals should look like white noise: no pattern, constant variance, roughly normal.
    • Check ACF of residuals and Ljung–Box test (no significant autocorrelation).
    • If residuals show structure → re‑specify orders, re‑difference, or consider seasonality/exogenous variables.

6) Forecasting

  • Once fitted, the model produces h‑step ahead forecasts and prediction intervals.
  • Forecasts from differenced models are integrated back (summed) to the original scale.
  • Uncertainty widens with horizon because future shocks accumulate.
  • A generic interval looks like $\hat{x}_{t+h|t} \pm z_{\alpha/2}\,\text{SE}_h$​ (assuming approximate normality of forecast errors).

7) Seasonal and exogenous extensions

  • SARIMA or ARIMA with seasonality:
    • $\text{SARIMA}(p,d,q)\,(P,D,Q)_s$s adds seasonal AR/MA terms and seasonal differencing of period $s$ (e.g., $s=12$ for monthly data).
  • The full form is:
    • $\Phi(B^s)\,\phi(B)\,(1-B)^d(1-B^s)^D x_t = c + \Theta(B^s)\,\theta(B)\,\varepsilon_t.$
    • Often you do seasonal differencing first if seasonality is strong.
  • ARIMAX / SARIMAX: include exogenous regressors $X_t$​ to explain part of the variation:
    • $\phi(B)\,(1-B)^d x_t = c + \beta^\top X_t + \theta(B)\,\varepsilon_t.$
    • Useful when promotions, prices, holidays, or weather help prediction.

8) Choosing orders and comparing models

  • Fit multiple candidates and compare AIC/AICc/BIC (lower is better, with AICc preferred for short samples).
  • Also compare out‑of‑sample accuracy (e.g., rolling origin cross‑validation) because information criteria don’t guarantee best forecast performance.
  • Automated tools (e.g., “auto.arima”‑style algorithms) apply unit‑root tests + stepwise search to minimize AICc, but diagnostics still matter.

9) Strengths and limits

Strengths

  • Lean, interpretable, fast to fit; strong baselines on many “short‑memory” series.
  • Works well when structure is mostly linear, with stable autocorrelations.

Limits

  • Handles one series at a time (univariate).
  • Struggles with strong nonlinearities, regime changes, heavy outliers, or time‑varying parameters.
  • Needs care for multiple seasonalities (e.g., hourly data with daily and weekly cycles); consider SARIMA, TBATS, Prophet, or state‑space models.

10) Common pitfalls (and fixes)

  • Over‑differencing: ACF of differenced series shows negative spike at lag 1 and excessive noise → reduce $d$ (or $D$).
  • Ignoring seasonality: Strong seasonal spikes in ACF at multiples of $s$ → include seasonal differencing ($D$) and seasonal AR/MA terms.
  • Residual autocorrelation: Ljung–Box significant → increase $p$/$q$ (or seasonal orders), revisit differencing, or add relevant exogenous variables.
  • Non‑constant variance: Apply a stabilizing transform (e.g., log) before differencing.
  • Parameter non‑invertibility/non‑stationarity: Refit with constraints; ensure roots lie outside the unit circle.

11) A quick practical workflow (checklist)

  1. Plot the series; consider log/Box‑Cox if variance grows with the level.
  2. Test/inspect for unit roots; apply minimal nonseasonal and seasonal differencing to achieve stationarity.
  3. Use ACF/PACF to propose candidate $(p,d,q)$ (and seasonal $(P,D,Q)_s$​).
  4. Estimate by MLE; check AIC/AICc/BIC.
  5. Perform residual diagnostics (plots, Ljung–Box, normality).
  6. Compare forecast accuracy with rolling/held‑out evaluation.
  7. Produce forecasts and prediction intervals; sanity‑check against domain knowledge.

Bottom line

ARIMA is a disciplined, interpretable way to turn a single time series into reliable forecasts by (i) removing non‑stationary behavior via differencing, (ii) modeling remaining autocorrelation through AR and MA terms, and (iii) validating with rigorous residual diagnostics. When seasonality or external drivers matter, extend it to SARIMA or ARIMAX/SARIMAX.