LSTM — Long Short-Term Memory Networks

An LSTM is a recurrent network with a memory cell and gates that decide what to keep, what to add, and what to output at each step. Hochreiter and Schmidhuber introduced it in 1997 to address a specific failure of plain recurrent networks: gradients that shrink to nothing or blow up as they travel back through time.

One piece of history is worth getting right, when comparing historical and modern equations. The 1997 architecture had an input gate and an output gate and no forget gate. The forget gate arrived three years later, in Gers, Schmidhuber and Cummins' “Learning to Forget” (Neural Computation 12(10), 2000), which is also where the equation everyone writes as the original comes from.

Why recurrent gradients can fade

A scalar example isolates the difficulty: a linear recurrence with weight \(w\) contributes \(w^T\) over \(T\) steps. Values 0.9 and 1.1 give decay and growth, while 1 gives a constant derivative. A nonlinear RNN also multiplies by activation derivatives and, in multiple dimensions, Jacobian matrices. A recurrent weight above one therefore does not by itself prove exploding gradients or divergent training.

An LSTM has both a cell state C, which carries retained information, and a hidden state h, which exposes an output to the next step or prediction head. Along the direct cell-state path, holding gate values fixed, the retained contribution is a product of forget gates. At the limiting value \(f=1\), the old cell contribution is preserved; a nonzero input contribution can still change the cell state.

The update \(C_t=f_t\odot C_{t-1}+i_t\odot\tilde C_t\) adds a new contribution to retained state. The direct path has derivative \(f_t\) when gate and candidate values are held fixed. The full derivative also includes their dependence on earlier hidden states. Gates near one provide a route for retaining information, but do not guarantee successful long-range learning; exact one is an ideal limit for a sigmoid with finite real inputs.

For a constant forget gate of 0.95, the direct retained contribution after 200 steps is 0.95²⁰⁰, about 0.000035. At 0.99, the contribution after 100 steps is about 0.366; at 0.999, it is about 0.905. Long-range memory requires the gate to actually learn to stay near 1 on the relevant inputs. The architecture makes it possible, not automatic.

The gates, and what they cost

For the standard variant below, ⊙ means elementwise multiplication, brackets concatenate vectors, and C and h have the hidden width. Each step computes three gates and a candidate from the same two inputs, the previous hidden state \(h_{t-1}\) and the current input \(x_t\):

• \(f_t = \sigma(W_f [h_{t-1}, x_t] + b_f)\) — how much of the old cell state survives.
• \(i_t = \sigma(W_i [h_{t-1}, x_t] + b_i)\) — how much of the new candidate is written.
• \(\tilde{C}_t = \tanh(W_C [h_{t-1}, x_t] + b_C)\) — the candidate itself.
• \(o_t = \sigma(W_o [h_{t-1}, x_t] + b_o)\), giving \(h_t = o_t \odot \tanh(C_t)\) — what leaves the cell.

The gates use a sigmoid for a reason that is easy to miss: its output lies in \((0,1)\), so multiplying by it is a soft, differentiable version of “keep this fraction.” The candidate uses tanh to supply a bounded signed value, allowing either positive or negative additions. These are vector operations: different cell components can retain and update different information at the same step.

For a one-component example, suppose the old cell value is 2, the forget gate is 0.9, the input gate is 0.2, and the candidate is −0.5. The new cell is 0.9 × 2 + 0.2 × (−0.5) = 1.7. With output gate 0.5, the hidden output is 0.5 × tanh(1.7), about 0.468. The cell can retain a value even when the output gate exposes only part of it. Gate values are learned responses to inputs and previous hidden state, not hand-coded decisions about which words or events matter.

With input width d and hidden width H, four affine transformations with one combined bias vector each contain 4H(d + H + 1) parameters. For d = 64 and H = 128, that is 98,816. This counts three gates plus the candidate, not four gates. A framework storing separate input and recurrent bias vectors has a different count: the standard PyTorch unprojected layer stores 4H(d + H + 2), or 99,328 here. Projections, multiple layers, and bidirectionality add further parameters.

Where they still fit

Training unrolls the recurrence over sequence steps and backpropagates through that computation. Truncated backpropagation limits how far gradients travel to reduce memory and computation; retaining a cell value across chunks does not mean a later loss can train dependencies beyond a detached gradient boundary. Padding must be excluded from the intended sequence computation and loss, and state should be reset between unrelated sequences. Carrying state between unrelated users or records leaks context.

A unidirectional LSTM can update a fixed-size state as new observations arrive, which is useful when an application cannot wait for future inputs. Bidirectional LSTMs combine forward and backward context and therefore require the future part of the sequence before producing those representations. Choose this distinction according to the information available at prediction time, not just validation accuracy.

The hidden-state dependency limits parallel execution across time, although batches and matrix operations can be parallelized. Standard full self-attention in a transformer permits more parallel work over known sequence positions but has pairwise attention work that grows quadratically with length; attention variants and autoregressive generation have different constraints. Compare quality, state size, latency, training cost, and sequence requirements on the actual workload. An LSTM is not automatically smaller or better on a small dataset. Its gates use the same learned transformations and gradient machinery introduced in neural networks.

References

• Hochreiter, S., and Schmidhuber, J. (1997). Long Short-Term Memory. Neural Computation, 9(8), 1735–1780. doi.org
• Gers, F. A., Schmidhuber, J., and Cummins, F. (2000). Learning to Forget: Continual Prediction with LSTM. Neural Computation, 12(10), 2451–2471. doi.org
• PyTorch documentation, torch.nn.LSTM. docs.pytorch.org