Gradient Descent with Momentum

Gradient descent with momentum usually converges faster than standard gradient descent. Instead of updating parameters using only the current gradient, it maintains an exponentially weighted average of recent gradients and uses that smoothed direction for each update.

This reduces unproductive oscillation while preserving movement in directions where gradients remain consistent.

Why Standard Gradient Descent Can Be Slow

Consider a cost function with elongated contours. One direction is steep and narrow, while another direction extends gradually toward the minimum.

Standard gradient descent may repeatedly cross the narrow valley:\[ \text{one side} \rightarrow \text{other side} \rightarrow \text{one side} \rightarrow \cdots \]

Although the overall trajectory approaches the minimum, much of each step is spent moving back and forth instead of directly toward the optimum.

These oscillations create two problems:

• Convergence is slow.
• The learning rate cannot be increased safely.

If the learning rate is too large, the updates may overshoot the valley and diverge.

Different Directions Need Different Behavior

Suppose the vertical direction is steep and causes oscillation, while the horizontal direction points toward the minimum.

The desired behavior is:

• Smaller updates in the vertical direction
• Larger and more consistent updates in the horizontal direction

Ordinary gradient descent does not remember previous gradients. Every update depends only on the current derivative.

Momentum adds this memory.

Momentum dampens directions that repeatedly reverse while reinforcing directions that remain consistent.

The Momentum Equations

At iteration \(t\), first calculate the ordinary gradients:\[ dW_t,\qquad db_t \]

Then update the exponentially weighted averages:\[ V_{dW,t} = \beta V_{dW,t-1} + (1-\beta)dW_t \]\[ V_{db,t} = \beta V_{db,t-1} + (1-\beta)db_t \]

Use these averages to update the parameters:\[ W_t = W_{t-1} – \alpha V_{dW,t} \]\[ b_t = b_{t-1} – \alpha V_{db,t} \]

Here:

• \(\alpha\) is the learning rate.
• \(\beta\) controls the amount of momentum.
• \(V_{dW}\) and \(V_{db}\) are smoothed gradient estimates.

Applying Momentum to Every Layer

For a network with \(L\) layers, maintain separate moving averages for each parameter:\[ V_{dW}^{[1]},V_{db}^{[1]}, \ldots, V_{dW}^{[L]},V_{db}^{[L]} \]

At iteration \(t\):\[ V_{dW,t}^{[l]} = \beta V_{dW,t-1}^{[l]} + (1-\beta)dW_t^{[l]} \]\[ V_{db,t}^{[l]} = \beta V_{db,t-1}^{[l]} + (1-\beta)db_t^{[l]} \]

Then update:\[ W_t^{[l]} = W_{t-1}^{[l]} – \alpha V_{dW,t}^{[l]} \]\[ b_t^{[l]} = b_{t-1}^{[l]} – \alpha V_{db,t}^{[l]} \]

Why Momentum Reduces Oscillation

Imagine several recent gradients whose vertical components alternate:\[ +g_y,\ -g_y,\ +g_y,\ -g_y \]

Their average is close to zero:\[ \operatorname{average} \left( +g_y,-g_y,+g_y,-g_y \right) \approx0 \]

Now suppose their horizontal components consistently point toward the minimum:\[ g_x,\ g_x,\ g_x,\ g_x \]

Their average remains large:\[ \operatorname{average} \left( g_x,g_x,g_x,g_x \right) =g_x \]

The resulting momentum vector therefore has:

• A small oscillating component
• A large consistent component toward the minimum

This gives the optimizer a smoother, more direct trajectory.

Exponentially Weighted Gradient History

Expanding the momentum recurrence gives:\[ V_{dW,t} = (1-\beta)dW_t + (1-\beta)\beta dW_{t-1} + (1-\beta)\beta^2dW_{t-2} +\cdots \]

The most recent gradient receives the largest weight, while older gradients receive exponentially decreasing weights.

The approximate number of recent gradients being averaged is:\[ \frac{1}{1-\beta} \]

For:\[ \beta=0.9 \]

this is approximately:\[ \frac{1}{1-0.9}=10 \]

Momentum therefore behaves roughly like an average of the latest ten gradients.

Physical Intuition: A Ball Rolling Downhill

A common analogy treats optimization as a ball rolling down a bowl.

• The gradient acts like acceleration.
• The momentum variable acts like velocity.
• The parameter \(\beta\) introduces persistence and friction.

As gradients continue pointing in a similar direction, the ball gains speed. When gradients alternate across a narrow valley, opposing accelerations partially cancel.

Because:\[ 0\leq\beta<1 \]

the stored velocity gradually decays rather than increasing without limit.

This analogy is optional, but it captures the idea that momentum links the current update to previous directions.

Initialization

Initialize the velocity terms to zero:\[ V_{dW,0}^{[l]}=0 \]\[ V_{db,0}^{[l]}=0 \]

Their dimensions must match the gradients:\[ V_{dW}^{[l]} \text{ has the same shape as } dW^{[l]} \text{ and }W^{[l]} \]\[ V_{db}^{[l]} \text{ has the same shape as } db^{[l]} \text{ and }b^{[l]} \]

In Python:

import numpy as np


def initialize_velocity(parameters):
    velocity = {}
    num_layers = len(parameters) // 2

    for layer in range(1, num_layers + 1):
        velocity[f"dW{layer}"] = np.zeros_like(
            parameters[f"W{layer}"]
        )

        velocity[f"db{layer}"] = np.zeros_like(
            parameters[f"b{layer}"]
        )

    return velocity

import numpy as np


def initialize_velocity(parameters):
    velocity = {}
    num_layers = len(parameters) // 2

    for layer in range(1, num_layers + 1):
        velocity[f"dW{layer}"] = np.zeros_like(
            parameters[f"W{layer}"]
        )

        velocity[f"db{layer}"] = np.zeros_like(
            parameters[f"b{layer}"]
        )

    return velocity

Parameter Updates with Momentum

def update_with_momentum(
    parameters,
    gradients,
    velocity,
    learning_rate,
    beta=0.9,
):
    num_layers = len(parameters) // 2

    for layer in range(1, num_layers + 1):
        dW_key = f"dW{layer}"
        db_key = f"db{layer}"
        W_key = f"W{layer}"
        b_key = f"b{layer}"

        velocity[dW_key] = (
            beta * velocity[dW_key]
            + (1 - beta) * gradients[dW_key]
        )

        velocity[db_key] = (
            beta * velocity[db_key]
            + (1 - beta) * gradients[db_key]
        )

        parameters[W_key] -= (
            learning_rate
            * velocity[dW_key]
        )

        parameters[b_key] -= (
            learning_rate
            * velocity[db_key]
        )

    return parameters, velocity

def update_with_momentum(
    parameters,
    gradients,
    velocity,
    learning_rate,
    beta=0.9,
):
    num_layers = len(parameters) // 2

    for layer in range(1, num_layers + 1):
        dW_key = f"dW{layer}"
        db_key = f"db{layer}"
        W_key = f"W{layer}"
        b_key = f"b{layer}"

        velocity[dW_key] = (
            beta * velocity[dW_key]
            + (1 - beta) * gradients[dW_key]
        )

        velocity[db_key] = (
            beta * velocity[db_key]
            + (1 - beta) * gradients[db_key]
        )

        parameters[W_key] -= (
            learning_rate
            * velocity[dW_key]
        )

        parameters[b_key] -= (
            learning_rate
            * velocity[db_key]
        )

    return parameters, velocity

The velocity dictionary must persist across updates. Reinitializing it during every iteration would eliminate the accumulated momentum.

Using Momentum with Mini-Batches

Momentum is especially useful with mini-batch gradient descent because mini-batch gradients are noisy.

For each mini-batch:

1. Perform forward propagation.
2. Compute the mini-batch cost.
3. Perform backpropagation.
4. Update the moving averages.
5. Update the parameters using the moving averages.

velocity = initialize_velocity(parameters)

for epoch in range(num_epochs):
    mini_batches = create_mini_batches(
        X_train,
        Y_train,
        batch_size,
        shuffle=True,
    )

    for X_batch, Y_batch in mini_batches:
        predictions, caches = forward_propagation(
            X_batch,
            parameters,
        )

        gradients = backward_propagation(
            predictions,
            Y_batch,
            caches,
        )

        parameters, velocity = update_with_momentum(
            parameters,
            gradients,
            velocity,
            learning_rate,
            beta=0.9,
        )

velocity = initialize_velocity(parameters)

for epoch in range(num_epochs):
    mini_batches = create_mini_batches(
        X_train,
        Y_train,
        batch_size,
        shuffle=True,
    )

    for X_batch, Y_batch in mini_batches:
        predictions, caches = forward_propagation(
            X_batch,
            parameters,
        )

        gradients = backward_propagation(
            predictions,
            Y_batch,
            caches,
        )

        parameters, velocity = update_with_momentum(
            parameters,
            gradients,
            velocity,
            learning_rate,
            beta=0.9,
        )

The same method also works with batch gradient descent, where each mini-batch is the complete training set.

Choosing \(\beta\)

The most common value is:\[ \beta=0.9 \]

This value is widely effective and represents smoothing across roughly ten recent updates.

Smaller \(\beta\)

A smaller value gives more weight to the current gradient:\[ \beta\downarrow \Rightarrow \text{less smoothing} \]

The optimizer responds more quickly but behaves more like ordinary gradient descent.

Larger \(\beta\)

A larger value retains more history:\[ \beta\uparrow \Rightarrow \text{more smoothing} \]

The updates become smoother, but the optimizer may respond more slowly when the gradient direction changes.

Values of \(\beta\) can be tuned, but \(0.9\) is a strong default.

Is Bias Correction Necessary?

Because the velocity variables begin at zero, their early values are biased toward zero.

A bias-corrected estimate would be:\[ \hat{V}_{dW,t} = \frac{V_{dW,t}}{1-\beta^t} \]\[ \hat{V}_{db,t} = \frac{V_{db,t}}{1-\beta^t} \]

For momentum, this correction is often omitted. With \(\beta=0.9\), the moving average warms up relatively quickly, and the early reduction in update size is usually acceptable.

This creates a short, natural warm-up period during the first few iterations.

Two Common Momentum Conventions

There are two related formulas in use.

Normalized formulation

\[ V_t = \beta V_{t-1} + (1-\beta)g_t \]

This directly represents an exponentially weighted average.

Unnormalized formulation

\[ V_t = \beta V_{t-1} + g_t \]

The second version omits \(1-\beta\). Its velocity scale is approximately larger by:\[ \frac{1}{1-\beta} \]

Therefore, it usually requires a differently tuned learning rate.

Both formulations can work, but their learning-rate values are not directly comparable.

When comparing implementations, check the momentum equation before comparing learning rates.

The normalized formulation is easier to interpret because it preserves the standard exponentially weighted average.

Momentum Hyperparameters

Momentum introduces two main hyperparameters:\[ \alpha \]

and:\[ \beta \]

The learning rate controls the overall update magnitude, while \(\beta\) controls smoothing.

Because they interact, changing \(\beta\) may require adjusting \(\alpha\), especially when switching between normalized and unnormalized conventions.

Standard Gradient Descent Versus Momentum

The additional memory has the same order as the model parameters because each weight and bias needs a corresponding velocity value.

Common Implementation Mistakes

Reinitializing velocity every iteration

Incorrect:

for X_batch, Y_batch in mini_batches:
    velocity = initialize_velocity(parameters)

for X_batch, Y_batch in mini_batches:
    velocity = initialize_velocity(parameters)

Correct:

velocity = initialize_velocity(parameters)

for X_batch, Y_batch in mini_batches:
    parameters, velocity = update_with_momentum(
        parameters,
        gradients,
        velocity,
        learning_rate,
        beta,
    )

velocity = initialize_velocity(parameters)

for X_batch, Y_batch in mini_batches:
    parameters, velocity = update_with_momentum(
        parameters,
        gradients,
        velocity,
        learning_rate,
        beta,
    )

Updating with the raw gradient

Incorrect:

W -= learning_rate * dW

W -= learning_rate * dW

Correct:

W -= learning_rate * V_dW

W -= learning_rate * V_dW

Mixing conventions unintentionally

If the implementation changes from:\[ V_t=\beta V_{t-1}+(1-\beta)g_t \]

to:\[ V_t=\beta V_{t-1}+g_t \]

the learning rate should be reconsidered.

Using incompatible shapes

The velocity and gradient must have identical shapes:

assert velocity[dW_key].shape == gradients[dW_key].shape
assert velocity[db_key].shape == gradients[db_key].shape

assert velocity[dW_key].shape == gradients[dW_key].shape
assert velocity[db_key].shape == gradients[db_key].shape

Key Takeaway

Gradient descent with momentum smooths recent gradients:\[ V_{dW,t} = \beta V_{dW,t-1} + (1-\beta)dW_t \]\[ V_{db,t} = \beta V_{db,t-1} + (1-\beta)db_t \]

and updates parameters using the smoothed direction:\[ W_t=W_{t-1}-\alpha V_{dW,t} \]\[ b_t=b_{t-1}-\alpha V_{db,t} \]

Momentum reduces oscillation in inconsistent directions and accelerates progress in directions where gradients remain aligned.

A common default is:\[ \beta=0.9 \]

Momentum requires little additional computation and usually trains neural networks faster than standard gradient descent.