Bias Correction in Exponentially Weighted Averages

ByJu Yeon Eum

Exponentially weighted averages provide an efficient way to smooth noisy sequences. However, when the running average is initialized at zero, its first values can be much smaller than the observations they are intended to represent.

Bias correction compensates for this initialization effect, producing more accurate estimates during the early stages.

The Standard Exponentially Weighted Average

Given observations:\[ \theta_1,\theta_2,\ldots,\theta_t \]

the running average is:\[ V_t = \beta V_{t-1} + (1-\beta)\theta_t \]

with:\[ V_0=0 \]

When \(\beta\) is close to 1, the average is smooth because it retains substantial information from the past.

For example:\[ \beta=0.98 \]

corresponds roughly to averaging over:\[ \frac{1}{1-0.98}=50 \]

recent observations.

The Initialization Problem

Consider the first update:\[ V_1 = 0.98V_0 + 0.02\theta_1 \]

Because:\[ V_0=0 \]

we obtain:\[ V_1=0.02\theta_1 \]

Suppose the first temperature is:\[ \theta_1=40 \]

Then:\[ V_1=0.02(40)=0.8 \]

A value of 0.8 is clearly not a reasonable estimate of a temperature near 40.

The estimate is biased toward zero because the running average begins with no accumulated history.

Initializing the average at zero causes the first estimates to be artificially small, especially when \(\beta\) is close to 1.

The Second Update

The next value is:\[ V_2 = 0.98V_1 + 0.02\theta_2 \]

Substituting:\[ V_1=0.02\theta_1 \]

gives:\[ V_2 = 0.98(0.02\theta_1) + 0.02\theta_2 \]

Therefore:\[ V_2 = 0.0196\theta_1 + 0.02\theta_2 \]

The coefficients sum to:\[ 0.0196+0.02=0.0396 \]

rather than 1.

As a result, \(V_2\) is still much smaller than a properly normalized weighted average of the first two observations.

Why the Weights Initially Sum to Less Than One

Expanding the recurrence gives:\[ V_t = (1-\beta) \sum_{k=0}^{t-1} \beta^k\theta_{t-k} \]

The total weight assigned to the observations is:\[ (1-\beta) \sum_{k=0}^{t-1} \beta^k \]

Using the geometric-series formula:\[ \sum_{k=0}^{t-1}\beta^k = \frac{1-\beta^t}{1-\beta} \]

we obtain:\[ (1-\beta) \sum_{k=0}^{t-1} \beta^k = 1-\beta^t \]

Thus, the weights sum to:\[ 1-\beta^t \]

During the early iterations, this can be much smaller than 1.

For example, with \(\beta=0.98\):\[ 1-\beta^1=0.02 \]\[ 1-\beta^2=0.0396 \]\[ 1-\beta^{10}\approx0.183 \]

This explains why the uncorrected estimate begins near zero and rises slowly.

The Bias-Correction Formula

The corrected estimate is:\[ \hat{V}_t = \frac{V_t}{1-\beta^t} \]

Dividing by the total accumulated weight normalizes the coefficients so that they sum to 1.

Bias correction divides the running estimate by the total weight accumulated so far.

Bias Correction on the First Observation

For \(t=1\):\[ V_1=(1-\beta)\theta_1 \]

The corrected estimate is:\[ \hat{V}_1 = \frac{V_1}{1-\beta} \]

Substituting:\[ \hat{V}_1 = \frac{ (1-\beta)\theta_1 }{ 1-\beta } \]

Therefore:\[ \hat{V}_1=\theta_1 \]

So the first corrected average is exactly the first observation.

With \(\theta_1=40\):\[ \hat{V}_1=40 \]

instead of the uncorrected value:\[ V_1=0.8 \]

Bias Correction on the Second Observation

For:\[ \beta=0.98 \]

and:\[ t=2 \]

the correction denominator is:\[ 1-\beta^t = 1-0.98^2 \]

Since:\[ 0.98^2=0.9604 \]

we obtain:\[ 1-0.98^2=0.0396 \]

The corrected estimate is:\[ \hat{V}_2 = \frac{V_2}{0.0396} \]

Substituting the expanded expression for \(V_2\):\[ \hat{V}_2 = \frac{ 0.0196\theta_1 + 0.02\theta_2 }{ 0.0396 } \]

The normalized coefficients are:\[ \frac{0.0196}{0.0396} \approx0.495 \]

and:\[ \frac{0.02}{0.0396} \approx0.505 \]

They sum to approximately:\[ 0.495+0.505=1 \]

Thus, \(\hat{V}_2\) is a properly normalized weighted average of the first two observations.

Why the Correction Becomes Unnecessary Later

As \(t\) increases:\[ \beta^t\rightarrow0 \]

Therefore:\[ 1-\beta^t\rightarrow1 \]

and:\[ \hat{V}_t = \frac{V_t}{1-\beta^t} \approx V_t \]

The corrected and uncorrected estimates eventually become almost identical.

For example, with \(\beta=0.98\):\[ 0.98^{100}\approx0.133 \]

so:\[ 1-0.98^{100}\approx0.867 \]

After 500 steps:\[ 0.98^{500}\approx0.000041 \]

and:\[ 1-0.98^{500}\approx0.999959 \]

At that point, the correction has almost no effect.

Corrected and Uncorrected Behavior

Without bias correction:

  • The moving average begins near zero.
  • It gradually rises toward the scale of the observations.
  • Early estimates are systematically too small.

With bias correction:

  • The first estimate begins at the first observation.
  • Early values form properly normalized weighted averages.
  • The initial warm-up distortion is removed.
  • Later values converge to the uncorrected average.

Python Implementation

def corrected_weighted_average(
    values,
    beta,
):
    average = 0.0
    corrected_values = []

    for t, value in enumerate(values, start=1):
        average = (
            beta * average
            + (1 - beta) * value
        )

        corrected_average = (
            average
            / (1 - beta**t)
        )

        corrected_values.append(
            corrected_average
        )

    return corrected_values

A version that returns both sequences is:

def weighted_averages(
    values,
    beta,
):
    average = 0.0

    uncorrected = []
    corrected = []

    for t, value in enumerate(values, start=1):
        average = (
            beta * average
            + (1 - beta) * value
        )

        bias_corrected = (
            average
            / (1 - beta**t)
        )

        uncorrected.append(average)
        corrected.append(bias_corrected)

    return uncorrected, corrected

Example

temperatures = [
    40.0,
    45.0,
    43.0,
]

uncorrected, corrected = weighted_averages(
    temperatures,
    beta=0.98,
)

print("Uncorrected:", uncorrected)
print("Corrected:", corrected)

The uncorrected values begin very close to zero, while the corrected values remain close to the scale of the observed temperatures.

Vector and Matrix Values

Bias correction works the same way for vectors and matrices.

Suppose \(G_t\) is a gradient matrix:\[ V_t = \beta V_{t-1} + (1-\beta)G_t \]

The corrected value is:\[ \hat{V}_t = \frac{V_t}{1-\beta^t} \]

In NumPy:

V = np.zeros_like(gradient)

V = (
    beta * V
    + (1 - beta) * gradient
)

V_corrected = V / (1 - beta**t)

The denominator is a scalar, so it is applied to every element.

Why Bias Correction Matters in Optimization

Optimization algorithms may use exponentially weighted averages of:

  • Gradients
  • Squared gradients
  • Parameter-update directions

At the beginning of training, these moving averages are initialized at zero. Without correction, the estimates can be artificially small.

Bias correction is particularly relevant when:

  • \(\beta\) is very close to 1.
  • Early iterations matter.
  • Training is relatively short.
  • The optimizer relies strongly on the scale of its moving averages.
  • Multiple moving averages interact in one update rule.

Algorithms such as Adam commonly include bias correction for both first- and second-moment estimates.

Why It Is Sometimes Omitted

In simpler applications, bias correction may be skipped because:

  • The warm-up period is short relative to total training.
  • The correction becomes negligible later.
  • A slightly biased initial estimate is acceptable.
  • Simpler implementation is preferred.

In these cases, the algorithm waits for the moving average to warm up naturally.

However, when accurate early estimates are important, the correction is inexpensive and useful.

Common Implementation Mistakes

Using the wrong exponent

The denominator must use the current step \(t\):\[ 1-\beta^t \]

not:\[ 1-\beta \]

for every iteration.

Starting \(t\) at zero

At \(t=0\):\[ 1-\beta^0=0 \]

which causes division by zero.

The first observation should use:\[ t=1 \]

Correcting the recurrence itself

First calculate the ordinary running value:\[ V_t = \beta V_{t-1} + (1-\beta)\theta_t \]

Then compute a separate corrected estimate:\[ \hat{V}_t = \frac{V_t}{1-\beta^t} \]

The corrected value should not normally replace \(V_t\) inside the next recurrence.

Correct:

average = (
    beta * average
    + (1 - beta) * value
)

corrected = average / (1 - beta**t)

The next update uses average, not corrected.

Key Takeaway

Initializing an exponentially weighted average with:\[ V_0=0 \]

causes early estimates to be biased toward zero because the accumulated weights sum to:\[ 1-\beta^t \]

rather than 1.

Bias correction removes this effect:\[ \hat{V}_t = \frac{V_t}{1-\beta^t} \]

Bias correction is most important during the early iterations, when the moving average has not yet accumulated enough history.

As \(t\) grows, \(\beta^t\) approaches zero and the correction becomes negligible. This technique provides more accurate early estimates and becomes especially important in optimization algorithms that rely on multiple exponentially weighted averages.

Similar Posts

Leave a Reply