RMSprop: Faster and More Stable Gradient Descent
RMSprop—short for Root Mean Square Propagation—is an optimization algorithm that adjusts the learning rate separately for each parameter.
It tracks an exponentially weighted average of squared gradients. Parameters with consistently large gradients receive smaller updates, while parameters with smaller gradients can continue moving more quickly.
This helps reduce oscillation, supports larger learning rates, and often accelerates neural-network training.
The Problem: Uneven Cost-Function Geometry
Consider a cost function with elongated contours. The surface is steep in one direction and relatively shallow in another.
For intuition, suppose:
- The horizontal axis represents a parameter \(w\).
- The vertical axis represents a parameter \(b\).
Standard gradient descent may oscillate strongly in the vertical direction while making slow progress horizontally toward the minimum.
The desired behavior is:
- Reduce updates in the steep, oscillating direction.
- Preserve or accelerate progress in the flatter direction.
RMSprop automatically scales each parameter’s update according to the recent magnitude of its gradients.
Standard Gradient Descent
At iteration \(t\), standard gradient descent calculates:\[ dW_t,\qquad db_t \]
and updates:\[ W_t=W_{t-1}-\alpha dW_t \]\[ b_t=b_{t-1}-\alpha db_t \]
The same global learning rate \(\alpha\) is applied to every parameter.
If gradients are much larger in one direction, that direction can dominate the updates and create severe oscillations.
RMSprop’s Squared-Gradient Averages
RMSprop maintains exponentially weighted averages of squared gradients:\[ S_{dW,t} = \beta_2S_{dW,t-1} + (1-\beta_2)dW_t^2 \]\[ S_{db,t} = \beta_2S_{db,t-1} + (1-\beta_2)db_t^2 \]
The squaring operations are element-wise:\[ dW_t^2=dW_t\odot dW_t \]\[ db_t^2=db_t\odot db_t \]
where \(\odot\) denotes element-wise multiplication.
Because the gradients are squared, positive and negative values do not cancel. Each running average measures the recent magnitude of the corresponding gradient components.
RMSprop Parameter Updates
After updating the squared-gradient averages, RMSprop changes the parameters using:\[ W_t = W_{t-1} – \alpha \frac{dW_t} {\sqrt{S_{dW,t}}+\varepsilon} \]\[ b_t = b_{t-1} – \alpha \frac{db_t} {\sqrt{S_{db,t}}+\varepsilon} \]
All division and square-root operations are element-wise.
The small constant \(\varepsilon\) prevents division by zero and improves numerical stability.
A common value is:\[ \varepsilon=10^{-8} \]
Why Squared Gradients Help
Suppose the cost function is much steeper in the vertical direction than in the horizontal direction.
Then the corresponding gradients may satisfy:\[ |db|\gg|dW| \]
Squaring them gives:\[ db^2\gg dW^2 \]
Therefore:\[ S_{db}\gg S_{dW} \]
During the update, the vertical gradient is divided by:\[ \sqrt{S_{db}} \]
which is relatively large. This substantially reduces the vertical update.
The horizontal gradient is divided by:\[ \sqrt{S_{dW}} \]
which is smaller, so horizontal progress remains comparatively strong.
RMSprop dampens directions with consistently large gradients while preserving movement in directions with smaller gradients.
Element-Wise Adaptive Scaling
The labels \(W\) and \(b\) are only a simplified illustration. A real neural network operates in a high-dimensional parameter space.
Some components of \(W\) may need strong damping, while others may not. RMSprop handles this component by component:\[ \left(S_{dW}\right)_{ij} = \beta_2 \left(S_{dW,\text{previous}}\right)_{ij} + (1-\beta_2) \left(dW_{ij}\right)^2 \]
Each weight therefore receives its own effective learning rate:\[ \alpha_{\text{effective},ij} = \frac{\alpha} {\sqrt{(S_{dW})_{ij}}+\varepsilon} \]
RMSprop does not literally use one rate for every matrix. It adapts the update scale for every individual parameter.
Why RMSprop Can Support a Larger Learning Rate
With standard gradient descent, a large learning rate may cause divergence in steep directions:\[ \alpha\uparrow \rightarrow \text{larger oscillations} \rightarrow \text{possible divergence} \]
RMSprop reduces those problematic updates through division by the root mean square of recent gradients.
This can make a larger global learning rate practical:\[ \text{large gradient magnitude} \rightarrow \text{large denominator} \rightarrow \text{smaller effective step} \]
The optimizer can then move more aggressively in useful directions without becoming as unstable in steep ones.
The Meaning of “Root Mean Square”
The name describes the essential calculation:
- Square the gradients.
- Compute an exponentially weighted mean.
- Take the square root.
- Divide the current gradient by that value.
Conceptually:\[ \operatorname{RMS}(g) \approx \sqrt{ \text{weighted average of }g^2 } \]
The update is therefore approximately:\[ \Delta\theta_t = -\alpha \frac{g_t} {\operatorname{RMS}(g)+\varepsilon} \]
Initializing RMSprop
For every layer, initialize the accumulators with zeros:\[ S_{dW,0}^{[l]}=0 \]\[ S_{db,0}^{[l]}=0 \]
Their shapes must match the corresponding parameters:\[ S_{dW}^{[l]} \text{ has the same shape as } W^{[l]} \]\[ S_{db}^{[l]} \text{ has the same shape as } b^{[l]} \]
import numpy as np
def initialize_rmsprop(parameters):
squared_gradients = {}
num_layers = len(parameters) // 2
for layer in range(1, num_layers + 1):
squared_gradients[f"dW{layer}"] = np.zeros_like(
parameters[f"W{layer}"]
)
squared_gradients[f"db{layer}"] = np.zeros_like(
parameters[f"b{layer}"]
)
return squared_gradientsImplementing RMSprop
def update_with_rmsprop(
parameters,
gradients,
squared_gradients,
learning_rate,
beta2=0.999,
epsilon=1e-8,
):
num_layers = len(parameters) // 2
for layer in range(1, num_layers + 1):
W_key = f"W{layer}"
b_key = f"b{layer}"
dW_key = f"dW{layer}"
db_key = f"db{layer}"
squared_gradients[dW_key] = (
beta2 * squared_gradients[dW_key]
+ (1 - beta2)
* np.square(gradients[dW_key])
)
squared_gradients[db_key] = (
beta2 * squared_gradients[db_key]
+ (1 - beta2)
* np.square(gradients[db_key])
)
parameters[W_key] -= (
learning_rate
* gradients[dW_key]
/ (
np.sqrt(squared_gradients[dW_key])
+ epsilon
)
)
parameters[b_key] -= (
learning_rate
* gradients[db_key]
/ (
np.sqrt(squared_gradients[db_key])
+ epsilon
)
)
return parameters, squared_gradientsThe accumulators must persist across updates. Reinitializing them for every mini-batch would erase the gradient history.
Using RMSprop with Mini-Batches
squared_gradients = initialize_rmsprop(
parameters
)
for epoch in range(num_epochs):
mini_batches = create_mini_batches(
X_train,
Y_train,
batch_size,
shuffle=True,
)
for X_batch, Y_batch in mini_batches:
predictions, caches = forward_propagation(
X_batch,
parameters,
)
gradients = backward_propagation(
predictions,
Y_batch,
caches,
)
parameters, squared_gradients = (
update_with_rmsprop(
parameters,
gradients,
squared_gradients,
learning_rate,
beta2=0.999,
epsilon=1e-8,
)
)RMSprop can also be used with batch gradient descent, but it is particularly helpful with noisy mini-batch gradients.
Choosing \(\beta_2\)
The parameter \(\beta_2\) determines how quickly the squared-gradient average responds.
A smaller value:
- Gives more influence to the current squared gradient
- Adapts more quickly
- Produces a noisier estimate
A larger value:
- Retains more historical information
- Produces smoother scaling
- Responds more slowly
Values near:\[ 0.9 \]
or:\[ 0.999 \]
may be encountered depending on the precise optimizer and convention. When RMSprop is incorporated into Adam, the standard second-moment default is commonly:\[ \beta_2=0.999 \]
The Role of \(\varepsilon\)
Without \(\varepsilon\), an accumulator that is zero or extremely small could cause division by zero or an excessively large update:\[ \frac{dW}{\sqrt{S_{dW}}} \]
RMSprop therefore uses:\[ \frac{dW} {\sqrt{S_{dW}}+\varepsilon} \]
A typical choice is:\[ \varepsilon=10^{-8} \]
Its purpose is numerical stability rather than meaningful regularization.
RMSprop Versus Momentum
Momentum and RMSprop both reduce problematic oscillation, but they do so differently.
Momentum
Momentum averages the gradients themselves:\[ V_t = \beta_1V_{t-1} + (1-\beta_1)g_t \]
It smooths the update direction and builds velocity along consistently aligned gradients.
RMSprop
RMSprop averages squared gradients:\[ S_t = \beta_2S_{t-1} + (1-\beta_2)g_t^2 \]
It adapts the update size separately for each parameter.
| Property | Momentum | RMSprop |
|---|---|---|
| Tracks | Gradients | Squared gradients |
| Main effect | Smooths direction | Scales step sizes |
| Reduces oscillation | Yes | Yes |
| Adaptive by parameter | Indirectly | Directly |
| Uses square root | No | Yes |
These two ideas can be combined, which leads to the Adam optimizer.
Bias Correction
Because:\[ S_0=0 \]
the initial squared-gradient averages are biased toward zero.
A bias-corrected estimate would be:\[ \hat{S}_t = \frac{S_t}{1-\beta_2^t} \]
Basic RMSprop implementations may omit this correction and allow the estimate to warm up naturally. Adam normally applies bias correction to both its gradient average and squared-gradient average.
Common Implementation Mistakes
Squaring the whole norm instead of each element
Incorrect:
S_dW = (
beta2 * S_dW
+ (1 - beta2)
* np.linalg.norm(dW) ** 2
)This produces one scalar and loses parameter-specific scaling.
Correct:
S_dW = (
beta2 * S_dW
+ (1 - beta2) * np.square(dW)
)Forgetting the square root
Incorrect:
W -= learning_rate * dW / (S_dW + epsilon)Correct:
W -= (
learning_rate
* dW
/ (np.sqrt(S_dW) + epsilon)
)Reinitializing accumulators
The S_dW and S_db values must persist across all updates.
Using matrix multiplication
The squared-gradient update, division, and square root must all be element-wise.
Omitting \(\varepsilon\)
Always include a small stabilizing constant in the denominator.
Key Takeaway
RMSprop tracks exponentially weighted averages of squared gradients:\[ S_{dW,t} = \beta_2S_{dW,t-1} + (1-\beta_2)dW_t^2 \]\[ S_{db,t} = \beta_2S_{db,t-1} + (1-\beta_2)db_t^2 \]
It then scales each update:\[ W_t = W_{t-1} – \alpha \frac{dW_t} {\sqrt{S_{dW,t}}+\varepsilon} \]\[ b_t = b_{t-1} – \alpha \frac{db_t} {\sqrt{S_{db,t}}+\varepsilon} \]
RMSprop reduces updates in directions with consistently large gradients and preserves stronger progress in directions with smaller gradients.
This adaptive scaling dampens oscillation, can support a larger learning rate, and often trains neural networks faster than standard gradient descent.