Introduction to Exponentially Weighted Averages
Exponentially weighted averages are a foundation for several optimization algorithms that train neural networks faster than standard gradient descent.
Also known as exponentially weighted moving averages, they smooth noisy measurements while retaining recent trends. A single parameter, \(\beta\), controls the balance between smoothness and responsiveness.
A Noisy Temperature Sequence
Suppose \(\theta_t\) represents the temperature recorded on day \(t\):\[ \theta_1,\theta_2,\ldots,\theta_{365} \]
Daily temperatures fluctuate because of short-term weather changes. However, the underlying seasonal pattern changes more gradually:
- Temperatures are generally lower near the beginning of the year.
- They rise toward summer.
- They decrease again toward the end of the year.
Plotting the raw observations produces a noisy curve. An exponentially weighted average smooths those fluctuations and reveals the broader trend.
The Basic Update Rule
Initialize:\[ V_0=0 \]
Then update the average for each observation:\[ V_t = \beta V_{t-1} + (1-\beta)\theta_t \]
where:
- \(V_t\) is the smoothed value at time \(t\).
- \(V_{t-1}\) is the previous smoothed value.
- \(\theta_t\) is the current observation.
- \(\beta\) determines how strongly the past is retained.
- \(1-\beta\) determines how strongly the current observation influences the result.
For:\[ \beta=0.9 \]
the update becomes:\[ V_t = 0.9V_{t-1} + 0.1\theta_t \]
The current estimate combines 90% of the previous estimate with 10% of the current temperature.
Step-by-Step Example
The first few updates are:\[ V_1 = 0.9V_0+0.1\theta_1 \]
Because \(V_0=0\):\[ V_1=0.1\theta_1 \]
The second update is:\[ V_2 = 0.9V_1+0.1\theta_2 \]
The third is:\[ V_3 = 0.9V_2+0.1\theta_3 \]
This process continues through the sequence.
Each new value combines the historical estimate with the latest observation.
Why It Smooths the Data
The raw sequence may change sharply from one observation to the next. The exponentially weighted average responds only partially to each new value.
If today’s temperature is unusually high, it affects the average, but it does not completely replace the accumulated history.
This reduces sensitivity to:
- Short-term fluctuations
- Measurement noise
- Isolated outliers
- Random variation
Exponentially weighted averaging reveals the underlying trend by blending recent observations with accumulated history.
The Approximate Averaging Window
A useful rule of thumb is:\[ \text{effective window} \approx \frac{1}{1-\beta} \]
This describes approximately how many recent observations have substantial influence.
It is not an exact moving window. Older values do not suddenly receive zero weight; their influence decays gradually.
Example: \(\beta=0.9\)
For:\[ \beta=0.9 \]
the approximate window is:\[ \frac{1}{1-0.9}=10 \]
This can be interpreted as averaging roughly the latest 10 observations.
The resulting curve typically provides moderate smoothing while remaining reasonably responsive to changes.
Example: \(\beta=0.98\)
For:\[ \beta=0.98 \]
the effective window is:\[ \frac{1}{1-0.98}=50 \]
This behaves roughly like an average over the latest 50 observations.
The update equation is:\[ V_t = 0.98V_{t-1} + 0.02\theta_t \]
Only 2% of the current value enters directly, while 98% comes from the previous estimate.
This produces a smoother curve, but it also introduces more delay.
Why a Large \(\beta\) Creates Lag
When \(\beta\) is close to 1, the historical estimate dominates:\[ \beta V_{t-1} \]
The current observation receives only a small coefficient:\[ (1-\beta)\theta_t \]
If the temperature begins rising quickly, the moving average takes time to catch up. If it begins falling, the average remains elevated for a while.
Therefore:\[ \beta\uparrow \Rightarrow \text{more smoothing} + \text{more lag} \]
A large \(\beta\) produces a stable estimate, but it adapts slowly when the underlying trend changes.
Example: \(\beta=0.5\)
For:\[ \beta=0.5 \]
the effective window is:\[ \frac{1}{1-0.5}=2 \]
The update becomes:\[ V_t = 0.5V_{t-1} + 0.5\theta_t \]
The current observation has a strong influence, so the result adapts quickly.
However, it is also:
- Noisier
- More sensitive to outliers
- Less effective at revealing a smooth long-term trend
Thus:\[ \beta\downarrow \Rightarrow \text{less smoothing} + \text{faster response} \]
Comparing Different Values of \(\beta\)
| \(\beta\) | Approximate window | Smoothness | Responsiveness | Lag |
|---|---|---|---|---|
| 0.5 | 2 observations | Low | High | Low |
| 0.9 | 10 observations | Moderate | Moderate | Moderate |
| 0.98 | 50 observations | High | Low | High |
There is no universally best value. The preferred setting depends on how much noise should be removed and how quickly the estimate should respond.
Python Implementation
def exponentially_weighted_average(
values,
beta,
):
average = 0.0
result = []
for value in values:
average = (
beta * average
+ (1 - beta) * value
)
result.append(average)
return resultExample:
temperatures = [
40.0,
48.0,
45.0,
43.0,
50.0,
]
smoothed = exponentially_weighted_average(
temperatures,
beta=0.9,
)
print(smoothed)The same function can be evaluated using different values:
short_average = exponentially_weighted_average(
temperatures,
beta=0.5,
)
medium_average = exponentially_weighted_average(
temperatures,
beta=0.9,
)
long_average = exponentially_weighted_average(
temperatures,
beta=0.98,
)Constant Memory Usage
An important advantage is that the complete history does not need to be stored.
Only two values are required at each step:
- The previous average \(V_{t-1}\)
- The current observation \(\theta_t\)
The running implementation is:
average = 0.0
for value in stream:
average = (
beta * average
+ (1 - beta) * value
)For a scalar sequence, the memory requirement is:\[ O(1) \]
This makes exponentially weighted averages suitable for long sequences and streaming data.
Why This Is Useful in Neural-Network Optimization
Mini-batch gradients can vary significantly between updates:\[ dW_1,dW_2,dW_3,\ldots \]
Rather than responding fully to every noisy gradient, an optimizer can track a smoothed estimate:\[ V_{dW,t} = \beta V_{dW,t-1} + (1-\beta)dW_t \]
The same idea can be applied to:
- Bias gradients
- Squared gradients
- Parameter-update directions
This produces updates that reflect recent trends rather than individual noisy measurements.
Exponentially Weighted Average Versus Exact Moving Average
An exact moving average over the latest \(k\) observations is:\[ M_t = \frac{1}{k} \sum_{i=0}^{k-1} \theta_{t-i} \]
This requires maintaining a window of recent values.
An exponentially weighted average instead uses:\[ V_t = \beta V_{t-1} + (1-\beta)\theta_t \]
The two methods differ in how they weight history:
- A moving average assigns equal weight within a fixed window.
- An exponentially weighted average assigns gradually decreasing weights to older observations.
The exponential method is especially attractive because it is simple, memory-efficient, and easy to update online.
Initialization Effect
Because:\[ V_0=0 \]
the first few estimates can be biased toward zero, especially when \(\beta\) is close to 1.
For example, with:\[ \beta=0.98 \]
the first update is:\[ V_1=0.02\theta_1 \]
which is much smaller than \(\theta_1\).
This initial bias can be corrected using a bias-correction term, which adjusts for the fact that the accumulated weights do not yet sum to approximately 1.
Key Takeaway
Exponentially weighted averages use the recursive update:\[ V_t = \beta V_{t-1} + (1-\beta)\theta_t \]
The parameter \(\beta\) controls the balance between smoothness and responsiveness:
A larger \(\beta\) creates a smoother estimate with greater lag.
A smaller \(\beta\) reacts faster but remains noisier.
The effective averaging window is approximately:\[ \frac{1}{1-\beta} \]
This simple mechanism becomes an important building block for optimization algorithms that smooth gradients and accelerate neural-network training.