RNN, GRU, and LSTM: A Complete Guide to Recurrent Networks

A recurrent neural network processes a sequence by carrying a hidden state from one timestep to the next. For a sentence, each input might be a word embedding; the state summarizes what the model has processed so far. Recurrence supports variable-length sequences with shared parameters. Feedforward architectures can also process sequences, but do not need to carry this recurrent state.

The recurrence

\[a^{\langle t\rangle}=g\left(W_{aa}a^{\langle t-1\rangle}+W_{ax}x^{\langle t\rangle}+b_a\right)\]

Here \(x^{\langle t\rangle}\) is the current input, \(a^{\langle t\rangle}\) the hidden state, and \(g\) an elementwise activation such as tanh. With input width \(n_x\) and hidden width \(n_a\), \(W_{aa}\) has shape \((n_a,n_a)\), \(W_{ax}\) has shape \((n_a,n_x)\), and the bias has \(n_a\) entries. The same weights are applied at every timestep, so their number is independent of sequence length. Responses can still depend on position through the accumulated state. For a scalar example with previous state 0.2, input 1, weights 0.5 and 0.3, and zero bias, the next state is \(\tanh(0.5\times0.2+0.3\times1)\approx0.380\).

import numpy as np

def rnn_forward(x, a0, Waa, Wax, ba):
    """x: (n_x, m, T_x). Returns all hidden states."""
    n_a, m = a0.shape
    T_x = x.shape[2]
    a = np.zeros((n_a, m, T_x))
    a_prev = a0
    for t in range(T_x):
        a_prev = np.tanh(Waa @ a_prev + Wax @ x[:, :, t] + ba)
        a[:, :, t] = a_prev
    return a

rng = np.random.default_rng(0)
x = rng.standard_normal((3, 4, 5))          # 3 features, 4 examples, 5 steps
a0 = np.zeros((6, 4))
out = rnn_forward(x, a0, rng.standard_normal((6, 6)) * 0.1,
                  rng.standard_normal((6, 3)) * 0.1, np.zeros((6, 1)))
print(out.shape)
# (6, 4, 5)

The NumPy example uses axes (features, batch, time) and starts each sequence at a zero state. The output stores six hidden features for each of four examples at five timesteps. A task head turns a state into a prediction, for example logits \(W_y a^{\langle t\rangle}+b_y\) for token tagging, or a head on the final valid state for sequence classification. Generation predicts a next token repeatedly. Encoder–decoder models can map an input sequence to a different-length output; the linked sequence-to-sequence article develops that construction.

How initialization affects signal and gradient propagation is explained in Weight Initialization and Gradient Flow in Deep Networks.

Backpropagation through time

Unrolling the network over \(T\) timesteps exposes a computation graph with repeated uses of the same weights. Backpropagation through time (BPTT) applies the chain rule to this graph. Let \(\delta_t=\partial\mathcal L/\partial z^{\langle t\rangle}\), where \(z\) is the pre-activation. This gradient includes the current output loss and any later losses reached through the state. For one sequence, the shared recurrent-weight gradient is:

\[\frac{\partial\mathcal L}{\partial W_{aa}}=\sum_{t=1}^T\delta_t\left(a^{\langle t-1\rangle}\right)^T.\]

The outer product at each step has shape \((n_a,n_a)\); batching adds contributions from examples as well. Ordinary full BPTT stores intermediate activations with memory growing linearly in sequence length at fixed batch and width. Checkpointing can trade storage for recomputation. Truncated BPTT detaches the carried state at chunk boundaries: its numerical value survives, but gradients cannot assign credit across that boundary. A model may still carry information farther than the chunk length, so truncation is not a hard limit on every dependency it can represent or learn indirectly.

Why gradients vanish over time

Write \(J_i=\operatorname{diag}(g'(z^{\langle i\rangle}))W_{aa}\). The contribution of a loss at step \(t\) to the gradient at an earlier state \(k\) is obtained in this order:

\[\nabla_{a^{\langle k\rangle}}\mathcal L_t=J_{k+1}^{T}J_{k+2}^{T}\cdots J_t^{T}\nabla_{a^{\langle t\rangle}}\mathcal L_t.\]

Each Jacobian depends on the state through the activation derivative. If every Jacobian has operator norm at most some \(q<1\), this contribution is bounded by \(q^{t-k}\) times its starting norm. A norm above 1 permits amplification but does not ensure that the actual gradient grows; directions and the ordered product matter. For tanh, derivatives lie between 0 and 1 and approach zero in saturation. The following scalar calculation isolates a chosen derivative factor of 0.25, with recurrent weight 1; it is not a measured typical value for an RNN.

deriv = 0.25          # chosen local derivative; recurrent weight 1
for steps in [5, 10, 20, 50]:
    print(steps, f"{deriv ** steps:.3e}")
# 5  9.766e-04
# 10 9.537e-07
# 20 9.095e-13
# 50 7.889e-31

Long-range language dependencies can be difficult to learn when these gradient contributions become very small. The scalar calculation illustrates attenuation; it cannot establish that a particular RNN will fail on a particular sentence.

Large gradient norms can destabilize updates before any NaN appears. Gradient clipping limits the update contribution of a finite gradient, but does not repair nonfinite values, guarantee stable training, or restore vanished gradients.

GRU: gates as a solution

Gated recurrences provide a direct state-retention path alongside a candidate update. Gates choose elementwise retention and writing coefficients from the current input and previous state. These paths can improve gradient propagation, but their coefficients still multiply over time.

A GRU has two gates, vectors with entries between 0 and 1 produced by sigmoid functions. In the convention below, the update gate \(\Gamma_u\) weights the new candidate, and the reset gate \(\Gamma_r\) controls which previous-state components enter that candidate. Here the sigmoid helper clips extreme pre-activations for this NumPy demonstration; it approximates the sigmoid outside that range. The parameter dictionary contains three matrices of shape \((n_a,n_a+n_x)\) and three biases of shape \((n_a,1)\); vstack joins state and input along the feature axis. Run the NumPy blocks in order.

def sigmoid(z):
    return 1 / (1 + np.exp(-np.clip(z, -30, 30)))

def gru_cell(x_t, a_prev, p):
    concat = np.vstack([a_prev, x_t])
    gamma_u = sigmoid(p["Wu"] @ concat + p["bu"])      # update gate
    gamma_r = sigmoid(p["Wr"] @ concat + p["br"])      # reset gate

    a_reset = np.vstack([gamma_r * a_prev, x_t])
    a_cand = np.tanh(p["Wa"] @ a_reset + p["ba"])

    return gamma_u * a_cand + (1 - gamma_u) * a_prev

p_gru = {"W"+k: np.zeros((1, 2)) for k in ("u", "r", "a")}
p_gru.update({"b"+k: np.zeros((1, 1)) for k in ("u", "r", "a")})
p_gru["bu"][:] = np.log(0.1 / 0.9)
p_gru["ba"][:] = np.arctanh(0.2)
print(round(gru_cell(np.zeros((1, 1)), np.array([[0.8]]), p_gru).item(), 3))
# 0.74

With old state 0.8, candidate 0.2, and update gate 0.1, the next state is \(0.1(0.2)+0.9(0.8)=0.74\). When the gate is near zero, retention is approximate, with direct-path derivative \(1-\Gamma_u\). The full derivative also includes changes in the gates and candidate. PyTorch uses an update gate that weights the old state, and applies its reset gate after the recurrent affine transform when forming the candidate. Thus this reset-before NumPy cell is not a drop-in numerical replica of nn.GRU.

LSTM: separate memory and output

An LSTM splits the state in two. The cell state \(c^{\langle t\rangle}\) is the long-term memory, and the hidden state \(a^{\langle t\rangle}\) is what the rest of the network sees. The cell below uses four affine branches on the concatenated input and hidden state, each with the same matrix and bias shapes as the GRU branches. Three gates control them: forget \(\Gamma_f\), input \(\Gamma_i\), and output \(\Gamma_o\).

\[c^{\langle t\rangle}=\Gamma_f\odot c^{\langle t-1\rangle}+\Gamma_i\odot\tilde c^{\langle t\rangle},\qquad a^{\langle t\rangle}=\Gamma_o\odot\tanh c^{\langle t\rangle}\]

The forget gate weights the old cell state; the input gate weights a tanh candidate; the output gate controls how much of the transformed cell state is exposed as the hidden state. Here \(\odot\) means elementwise multiplication. For \(c_{prev}=2,f=0.9,i=0.2,\tilde c=0.5\), the new cell value is \(0.9(2)+0.2(0.5)=1.9\). With \(o=0.5\), the hidden value is \(0.5\tanh(1.9)\approx0.478\). Unlike the complementary GRU mixing coefficients, LSTM retention and writing gates are independently parameterized. The direct cell-state path has coefficient \(f\), not automatically 1; products of forget gates can still decay. Additional gradient paths run through the hidden state and gates.

def lstm_cell(x_t, a_prev, c_prev, p):
    concat = np.vstack([a_prev, x_t])
    f = sigmoid(p["Wf"] @ concat + p["bf"])       # forget
    i = sigmoid(p["Wi"] @ concat + p["bi"])       # input
    o = sigmoid(p["Wo"] @ concat + p["bo"])       # output
    c_cand = np.tanh(p["Wc"] @ concat + p["bc"])

    c_next = f * c_prev + i * c_cand              # additive path
    a_next = o * np.tanh(c_next)
    return a_next, c_next

p_lstm = {"W"+k: np.zeros((1, 2)) for k in ("f", "i", "o", "c")}
p_lstm.update({"b"+k: np.zeros((1, 1)) for k in ("f", "i", "o", "c")})
p_lstm["bf"][:] = np.log(0.9 / 0.1)
p_lstm["bi"][:] = np.log(0.2 / 0.8)
p_lstm["bc"][:] = np.arctanh(0.5)
h, c = lstm_cell(np.zeros((1, 1)), np.zeros((1, 1)), np.array([[2.]]), p_lstm)
print(round(c.item(), 3), round(h.item(), 3))
# 1.9 0.478

A positive forget-gate bias is one initialization option supported by the experiments of Jozefowicz et al. When all other contributions to its pre-activation are zero, bias 0 gives \(\sigma(0)=0.5\), while bias 1 gives \(\sigma(1)\approx0.731\). Input and recurrent terms change the actual gate. This encourages initial retention but does not ensure long-term memory or improve every task. In PyTorch an effective bias is the sum of input and recurrent bias vectors; setting both forget slices to 1 would give an effective bias of 2.

Choosing between them

The table counts all weights in one unidirectional layer, not parameters per neuron. The NumPy cells add \(3n_a\) or \(4n_a\) biases; PyTorch normally has two bias vectors per branch, adding \(6n_a\) or \(8n_a\). GRU has fewer parameters at equal width, but runtime depends on kernels, hardware, and sequence sizes. Compare validation performance, latency, and memory on the intended task; neither architecture has a universal advantage for small datasets or long dependencies.

Bidirectional and deep recurrent networks

A bidirectional RNN processes the available sequence in both directions and commonly concatenates the two states at each position. The backward state can use future context, which is useful when the full input is available. It would leak future information in a strictly causal prediction task. A bidirectional encoder can still process a completed source sequence before an autoregressive decoder generates output; bounded-lookahead streaming is another, different latency setting.

A stacked RNN feeds each layer’s sequence of hidden states to the next layer. This adds transformations across layers as well as recurrence across time. Useful depth depends on the task and training setup; temporal unrolling does not establish a general three-layer limit.

Attention was introduced to fix a specific encoder-decoder failure, described in Sequence-to-Sequence Models and Attention Explained.

Practical checks

• Monitor gradient norms. If clipping is useful, apply it after backward and before the optimizer update, and validate the threshold; with scaled mixed-precision gradients, unscale first.
• Mask padded target positions in the loss. Packing additionally prevents padding from updating recurrent states; loss masking alone does not do that. For right-padded unidirectional outputs, gather the last valid timestep for sequence classification. Bidirectional recurrence especially needs correct length handling.
• Sort or bucket by length to reduce padding waste.
• For truncated BPTT, detach carried states at the intended chunk boundary (both hidden and cell states for LSTM). Carry them only when each batch slot continues the same stream; reset for unrelated sequences.
• PyTorch recurrent-module dropout acts between recurrent layers, excluding the final layer, so it has no effect with one layer. Recurrent dropout requires an explicitly chosen implementation; a mask fixed across time is one approach.
• Inspect gates alongside task performance and state changes. A forget gate near 1 can be retaining useful memory, while a gate near 0 can be intentionally resetting it. Saturation alone does not prove memory is unused.

Self-attention, masking, and the full block are derived in Transformers from Scratch: Self-Attention to Encoder–Decoder.

Training on variable-length sequences

This synthetic sequence-classification step connects recurrence to an output head and a loss. PyTorch with batch_first=True takes (batch, time, features), unlike the NumPy layout above. Lengths mark valid prefixes: the last two positions of the first example are excluded by packing even though the tensor contains numbers there. Packing requires positive lengths, and tensor lengths are kept on the CPU.

import torch
from torch import nn
from torch.nn.utils.rnn import pack_padded_sequence

torch.manual_seed(0)
sequence = torch.randn(2, 5, 3)
lengths = torch.tensor([3, 5])
labels = torch.tensor([0, 1])
rnn = nn.LSTM(input_size=3, hidden_size=4, batch_first=True)
head = nn.Linear(4, 2)
optimizer = torch.optim.SGD(list(rnn.parameters()) + list(head.parameters()), lr=0.1)
packed = pack_padded_sequence(sequence, lengths, batch_first=True, enforce_sorted=False)
_, (h_n, c_n) = rnn(packed)
logits = head(h_n[-1])
loss = nn.functional.cross_entropy(logits, labels)
optimizer.zero_grad()
loss.backward()
print(tuple(h_n.shape), tuple(c_n.shape), tuple(logits.shape))
# (1, 2, 4) (1, 2, 4) (2, 2)
print(bool(torch.isfinite(rnn.weight_ih_l0.grad).all()))
# True
optimizer.step()

For this single-layer, unidirectional LSTM, h_n and c_n have axes (layers, batch, hidden), and h_n[-1] contains each sequence’s final valid hidden state in original batch order. No states were supplied, so they started at zero. The classifier produces two logits per sequence; cross-entropy backpropagates through the classifier and recurrent weights. This checks a training step, not predictive quality. In general the state’s first axis is layers times directions; a bidirectional classifier needs the two directions’ final states, not just the last slice. An nn.GRU returns a hidden state without a separate cell state.

Exercises

1. A linear recurrence. Set the activation derivative to 1 and repeatedly apply \(W^T\) to a vector. Scale a fixed random matrix to spectral radii 0.9, 1.0, and 1.1, and compare norms after 50 steps. What does this experiment leave out of a tanh RNN?

You should get: substantially different finite-step norms, with no general boundedness conclusion at radius 1.

import numpy as np
rng = np.random.default_rng(0)
W0 = rng.normal(size=(64, 64))
W0 = W0 / max(abs(np.linalg.eigvals(W0)))               # spectral radius 1
for s in (0.9, 1.0, 1.1):
    g = np.ones(64)
    for _ in range(50):
        g = (s * W0).T @ g
    print(s, f"{np.linalg.norm(g):.3e}")
# 0.9 1.888e-02
# 1.0 3.663e+00
# 1.1 4.300e+02

2. Direct-path attenuation. Compare a scalar gradient contribution multiplied by 0.25 for 100 steps with the direct LSTM cell-state path under fixed forget gates of 0.99 and 0.5. Hold gate values fixed and ignore other paths. Report the remaining fraction; distinguish this calculation from a full recurrent-network gradient.

You should get: about 36.6% retention for 0.99, and much smaller positive fractions in the other cases.

print("tanh, deriv 0.25 ", f"{0.25**100:.3e}")     # 6.223e-61
for f in (0.99, 0.5):
    print(f"forget gate {f}   ", f"{f**100:.3e}")
# forget gate 0.99    3.660e-01
# forget gate 0.5     7.889e-31

3. Masking padded targets. Compute cross-entropy over sequences of length 3 and 10 padded to length 10, with and without excluding padding. Report both losses and the fraction of valid tokens. What training signal does the unmasked loss add?

You should get: 13 valid targets out of 20 positions; the masked average excludes seven padding targets.

import torch, torch.nn.functional as F
torch.manual_seed(0)
V, T = 5, 10
logits = torch.randn(2, T, V)
targets = torch.zeros(2, T, dtype=torch.long)
targets[0, :3] = torch.randint(1, V, (3,))     # real tokens
targets[1, :10] = torch.randint(1, V, (10,))
mask = targets != 0                            # 0 is the pad id

unmasked = F.cross_entropy(logits.reshape(-1, V), targets.reshape(-1))
masked = F.cross_entropy(logits.reshape(-1, V), targets.reshape(-1),
                         ignore_index=0)
print(round(unmasked.item(), 4), round(masked.item(), 4),
      "| real fraction", round(float(mask.float().mean()), 3))
# 1.5785 1.5457 | real fraction 0.65

References

• Bengio, Simard, and Frasconi (1994). Learning Long-Term Dependencies with Gradient Descent Is Difficult. IEEE Transactions on Neural Networks.
• Hochreiter and Schmidhuber (1997). Long Short-Term Memory. Neural Computation.
• Schuster and Paliwal (1997). Bidirectional Recurrent Neural Networks. IEEE Transactions on Signal Processing.
• Cho et al. (2014). Learning Phrase Representations Using RNN Encoder-Decoder for Statistical Machine Translation. EMNLP.
• Pascanu, Mikolov, and Bengio (2013). On the Difficulty of Training Recurrent Neural Networks. ICML.
• Jozefowicz, Zaremba, and Sutskever (2015). An Empirical Exploration of Recurrent Network Architectures. ICML.
• Gal and Ghahramani (2016). A Theoretically Grounded Application of Dropout in Recurrent Neural Networks. NIPS.
• PyTorch documentation: GRU, LSTM, and pack_padded_sequence.