Learning Rate Decay: Smaller Steps as Training Converges

Learning rate decay is a technique that gradually reduces the learning rate during training. It allows an optimization algorithm to make large, rapid updates initially and smaller, more precise updates as it approaches a minimum.

Why Reduce the Learning Rate?

Mini-batch gradient descent calculates each update using only a subset of the training data. Because different mini-batches produce slightly different gradients, the optimization path naturally contains noise.

With a fixed learning rate, the algorithm may approach a minimum but continue oscillating around it. It may never settle into a sufficiently small region.

Gradually reducing the learning rate helps balance two objectives:

• A large learning rate enables rapid progress during the early stages.
• A small learning rate reduces oscillation near convergence.
• The final parameter values remain within a tighter region around the minimum.

Learning rate decay lets the optimizer move quickly when it is far from the minimum and more carefully when it is close to the minimum.

Learning Rate Decay by Epoch

An epoch is one complete pass through the training set. A common learning rate schedule is:\[ \alpha_e = \frac{\alpha_0} {1+d e} \]

where:

• \(\alpha_0\) is the initial learning rate
• \(d\) is the decay rate
• \(e\) is the epoch number
• \(\alpha_e\) is the learning rate used during epoch \(e\)

The initial learning rate and decay rate are both hyperparameters.

Example

Suppose:\[ \alpha_0=0.2 \]

and:\[ d=1 \]

The resulting learning rates are:

The learning rate decreases relatively quickly at first and then continues declining more gradually.

initial_learning_rate = 0.2
decay_rate = 1.0

for epoch in range(5):
    learning_rate = initial_learning_rate / (
        1 + decay_rate * epoch
    )
    print(epoch, learning_rate)

initial_learning_rate = 0.2
decay_rate = 1.0

for epoch in range(5):
    learning_rate = initial_learning_rate / (
        1 + decay_rate * epoch
    )
    print(epoch, learning_rate)

Exponential Decay

Another common schedule reduces the learning rate exponentially:\[ \alpha_e=\alpha_0 r^e \]

where \(r\) is a number slightly smaller than \(1\), such as \(0.95\).

For example:\[ \alpha_e=0.2(0.95)^e \]

Each epoch multiplies the previous learning rate by the same decay factor.

initial_learning_rate = 0.2
decay_factor = 0.95

for epoch in range(5):
    learning_rate = initial_learning_rate * decay_factor**epoch
    print(epoch, learning_rate)

initial_learning_rate = 0.2
decay_factor = 0.95

for epoch in range(5):
    learning_rate = initial_learning_rate * decay_factor**epoch
    print(epoch, learning_rate)

A smaller decay factor reduces the learning rate more aggressively. A value closer to \(1\) produces slower decay.

Inverse Square-Root Decay

The learning rate can also decrease according to the inverse square root of the epoch number:\[ \alpha_e = \frac{k\alpha_0}{\sqrt{e}} \]

Because this expression is undefined at \(e=0\), it is often written in a safer form such as:\[ \alpha_e = \frac{\alpha_0}{\sqrt{1+ke}} \]

A related schedule can use the mini-batch update number \(t\):\[ \alpha_t = \frac{\alpha_0}{\sqrt{1+kt}} \]

Here, \(k\) controls the rate of decay.

Staircase Decay

Learning rates do not have to change continuously. A staircase schedule keeps the learning rate fixed for a certain number of epochs and then reduces it abruptly.

For example:\[ 0.2 \rightarrow 0.1 \rightarrow 0.05 \rightarrow 0.025 \]

Each reduction could occur after a predetermined number of epochs.

A general staircase schedule is:\[ \alpha_e = \alpha_0\gamma^{\left\lfloor e/s \right\rfloor} \]

where:

• \(\gamma\) is the decay factor
• \(s\) is the number of epochs between reductions
• \(\lfloor\cdot\rfloor\) is the floor operation

initial_learning_rate = 0.2
decay_factor = 0.5
step_size = 10

for epoch in range(40):
    learning_rate = (
        initial_learning_rate
        * decay_factor ** (epoch // step_size)
    )

initial_learning_rate = 0.2
decay_factor = 0.5
step_size = 10

for epoch in range(40):
    learning_rate = (
        initial_learning_rate
        * decay_factor ** (epoch // step_size)
    )

In this example, the learning rate is cut in half every ten epochs.

Manual Learning Rate Decay

The learning rate can also be adjusted manually by monitoring training progress. When improvement begins to slow or the optimization process starts oscillating, the learning rate can be reduced.

Manual control may be practical when:

• Only one or a few models are being trained.
• Each model takes many hours or days to train.
• Training is closely monitored.
• Automated experimentation is not required.

However, manually controlling the learning rate does not scale well when many models must be trained.

Comparing the Main Schedules

Choosing a Learning Rate Schedule

The initial learning rate usually has a greater effect on training performance than the exact decay schedule. It is therefore sensible to find a good fixed learning rate before spending substantial effort tuning decay behavior.

A practical process is:

1. Find an initial learning rate that produces stable and reasonably fast optimization.
2. Observe whether training begins to oscillate or stops improving near convergence.
3. Introduce a simple decay schedule.
4. Tune the decay rate only when it produces a meaningful improvement.

Learning rate decay is useful, but it is not always the first optimization technique that needs attention. A poorly chosen initial learning rate cannot usually be rescued by an elaborate decay schedule.

Key Takeaway

Learning rate decay combines fast initial progress with increasingly precise updates near convergence. It is especially useful with mini-batch optimization, where gradient noise can otherwise prevent the parameters from settling into a tight region around the minimum.