A Gentle Introduction to Stationarity

1) The Basic Time Series Modeling Paradigm

The Key Difficulty: One Observation per Time

In time series analysis, you start with observed data:$$x_1, x_2, \dots, x_n.$$

The modeling goal is to associate these observations with a stochastic process (a sequence of random variables):$$X_1, X_2, \dots, X_n,$$

where each $X_t$​ is a random variable that could have produced the observed value $x_t$​.

Conceptually, we would like each $X_t$​ to represent:

• all plausible values that could occur at time $t$, and
• a probability distribution describing how likely each value is.

But there is a fundamental problem: at each time $t$, you only observe one value $x_t$​.
From a single draw at each time, you cannot directly infer the full distribution of $X_t$​.

A helpful analogy illustrates this difficulty:
imagine having $n$ different biased coins.
If each coin is flipped only once, it is essentially impossible to determine the bias of any individual coin.

This highlights why time series modeling is hard if you treat each time step as a completely separate distribution.

---

2) Why Stationarity Helps

Stationarity (Preliminary Definition)

A process is stationary (roughly) if its statistical properties do not depend on time.

Informally, “the distribution looks the same today as it does tomorrow” in a probabilistic sense.

What “statistical properties” typically means:

• the mean does not change over time,
• the variance does not change over time,
• the way values depend on each other across time (correlation structure) does not change over time.

The critical benefit:

• If the process is stationary, then information from different time points can be pooled.
• A long time series acts like a “large dataset” for estimating the process’s behavior, because the distribution is assumed to be stable over time.

Even when observations are correlated across time (which is common), stationarity still makes modeling feasible because the dependence structure is also assumed to be time-invariant.

---

3) Getting to Stationarity: The Common Workflow

Transform → Model → Undo Transform

Many real-world series are not stationary. For example:

• an upward trend suggests the mean is increasing over time → not stationary.

So a common approach is:

1. Transform the original series $x_t$​ into a new series $y_t$​ that is closer to stationary
   (often called “residual-like” series after removing trend/seasonality).
2. Model $y_t$​ using a stationary stochastic process $Y_t$​.
3. Undo the transformation to map the stationary model back to a model for the original scale $X_t$.

This idea is central in classical time series methods: you try to isolate the “stationary core” because stationary processes are far easier to analyze and forecast with.

---

4) Why a Time Series Can Be Nonstationary (and What to Do)

A time series may be nonstationary for several common reasons:

1. Trend (long-term increase/decrease)
2. Seasonality (regular periodic pattern)
3. Structural breaks (sudden change in behavior)
4. Changing variance / scale (small variability earlier, large variability later)

Brief approaches for handling cases (3) and (4) are outlined below:

Structural breaks

If the series suddenly changes regime (e.g., policy change, economic shock), you may:

• split the series into segments and model each segment separately.

Changing variance / changing magnitude

If variability grows as the level grows, a common transformation is the log transform:$$\log(x_t)$$

If some $x_t \le 0$, you might use:$$\log(x_t + C)$$

for a constant $C$ large enough to make all terms positive.

This often stabilizes variance and makes patterns easier to model.

---

5) Classical Decomposition: Trend + Seasonality + Stationary Component

A common modeling assumption is the classical additive decomposition:$$X_t = m_t + s_t + Y_t$$

where:

• $m_t$​ is a deterministic trend function (slowly varying),
• $s_t$​ is a deterministic seasonal function (periodic),
• $Y_t$ is a stationary stochastic process capturing the remaining fluctuations.

The core idea:

• Identify and remove $m_t$​ and $s_t$​,
• then focus analysis on $Y_t$​, which is stationary.

Two major ways to remove trend/seasonality are mentioned:

1. regression/parameter fitting (explicitly estimate $m_t$​, $s_t$​),
2. differencing (the method emphasized here).

---

6) Differencing: The Main Tool Introduced

6.1 First difference (lag-1)

The first difference is:$$\nabla x_t = x_t – x_{t-1}$$

This often removes a trend.

In R, this is done with:

• diff(x) (default is lag 1)

---

6.2 Example: Lake Huron (Removing a Trend)

The Lake Huron annual water level series shows a downward trend.

Model idea:$$X_t = m_t + Y_t$$

(no seasonality; trend plus stationary fluctuations)

After differencing:$$\nabla X_t = \nabla m_t + \nabla Y_t$$

If $\nabla X_t$​ is stationary and $\nabla Y_t$​ is stationary, then $\nabla m_t$​ must be constant (not time-varying).
That implies $m_t$ must be linear:$$m_t = c_0 + c_1 t$$

Interpretation:

• If differencing once makes the series look stationary, you are effectively assuming the original trend is approximately linear.

An important clarification is the following:

• The differenced series $\nabla x_t$​ is associated with $\nabla X_t$​, not directly with the stationary component $Y_t$​.
  You can recover $Y_t$​ in principle, but that step was not performed here.

---

6.3 Higher-order differencing removes higher-order trends

If the trend is quadratic:$$m_t = c_0 + c_1 t + c_2 t^2$$

then differencing once produces something still trending (because the slope changes over time), but differencing twice removes the quadratic component:

Define second difference:$$\nabla^2 x_t = \nabla(\nabla x_t)$$

When $m_t$mt​ is a quadratic function, it can be shown that:

$$\nabla^2 m_t = 2c_2$$

which is constant.

General rule:

• differencing $k$ times removes a polynomial trend of degree $k$.

---

7) Seasonal Differencing (Lag-$d$ Differencing)

7.1 Lag-$d$ difference

To remove seasonality with period $d$, use:$$\nabla_d x_t = x_t – x_{t-d}$$

In R:

• diff(x, lag = d)

7.2 Example: Nottingham Temperatures (Monthly, strong yearly cycle)

Monthly temperatures typically have strong seasonality with period $12$.
So use lag-12 differencing:$$\nabla_{12} x_t = x_t – x_{t-12}$$

This subtracts “the value from the same month last year,” which often removes seasonal patterns.

---

7.3 Important notation warning: two very different operations

It is important not to confuse the following:

1. Repeated lag-1 differencing $\nabla^d$ (apply $\nabla$ $d$ times)
2. Single lag-$d$ differencing $\nabla_d$​ (apply a lag $d$ once)

Example when $d=2$:

• Repeated lag-1 differencing:

$$\nabla^2 x_t = x_t – 2x_{t-1} + x_{t-2}$$

• Lag-2 differencing:

$$\nabla_2 x_t = x_t – x_{t-2}$$

These are not the same and serve different modeling purposes:

• $\nabla^2$ targets polynomial trend curvature,
• $\nabla_2$​ targets seasonality of period 2.

---

8) Removing Multiple Nonstationary Features Together

Example: Johnson & Johnson Quarterly Earnings

This dataset shows:

• increasing variance (fluctuations become larger over time),
• trend,
• seasonality (quarterly cycle).

Step-by-step transformation strategy shown:

1. Apply log to stabilize variance:

$$\log(x_t)$$

2. Remove quarterly seasonality using lag-4 differencing:

$$\nabla_4 \log(x_t)$$

3. If needed, remove remaining trend by differencing once more:

$$\nabla(\nabla_4 \log(x_t))$$

A key modeling judgment issue is the following:

• The series after step 2 might already be “stationary enough.”
• Adding another differencing step increases model complexity.
• Deciding whether it is necessary requires more formal diagnostic tools and statistical tests, which are introduced later.

---

9) Why Differencing Is So Useful (Quick Review of Key Points)

• First differencing: $$\nabla x_t = x_t – x_{t-1}$$often removes trend.
• Repeated differencing can remove higher-order polynomial trends.
• Seasonal (lag-$d$) differencing: $$\nabla_d x_t = x_t – x_{t-d}$$removes seasonality with period $d$.
• These transformations can be combined (log + seasonal difference + regular difference).
• Differencing is reversible (information is not “lost” if you keep an initial value).
  For example, given $x_1$​ and the differenced series, you can reconstruct: $$x_t = x_1 + \sum_{s=2}^{t} \nabla x_s$$​ (same idea extends to stochastic processes).

---

Bottom line

Stationarity is introduced because it turns an otherwise impossible inference task (learning time-varying distributions from one observation per time) into a feasible one by assuming time-invariant statistical structure. When a series is nonstationary, you often transform it (log, differencing, seasonal differencing, segmentation) to isolate a stationary component that can be modeled and used for forecasting.