Sequence-to-Sequence Models and Attention Explained

Translation, summarization, and generative question answering can map an input sequence to an output of a different length. An encoder represents the input; a decoder uses that representation to generate output tokens. This article develops recurrent encoder–decoder models and attention, then distinguishes the training objective, decoding procedure, and evaluation metric.

Encoder and decoder

The encoder reads source tokens \(x\). The decoder models \(P(y\mid x)=\prod_{t=1}^{L}P(y_t\mid y_{<t},x)\), predicting each token from the source and previous target tokens. A beginning-of-sequence token (BOS) supplies the first decoder input. An end-of-sequence token (EOS) marks completion and is included in the sequence probability. Generation also needs a maximum-length limit in case EOS is never emitted.

An early recurrent design summarizes the source in a fixed-size final encoder state and uses it to initialize the decoder, sometimes through a learned projection. LSTM implementations must specify how both hidden and cell states are initialized. Such models can work, but compressing every source into the same-sized state can make detailed long-input reconstruction difficult.

The recurrent cells used by these models are covered in RNN, GRU, and LSTM: A Complete Guide to Recurrent Networks.

The bottleneck

A 50-token and a 5-token source use the same final-state width. In a forward recurrent encoder, early information passes through more updates before reaching that state. Input reversal, bidirectionality, model capacity, and training choices affect how severe this becomes.

Bahdanau et al. reported worse long-sentence translation in a fixed-vector baseline and improvements with attention in their experiment. A decline with length does not by itself isolate storage failure: data coverage, optimization, and decoding can also contribute. There is no universal 30-token failure boundary.

Attention

Attention removes the constraint by keeping every encoder hidden state and letting the decoder build a different summary at each output step. At step \(t\) the decoder computes a relevance score against each encoder state, normalizes the scores with a softmax, and takes the weighted average.

\[e_{t,i}=\text{score}(s_{t-1},h_i),\qquad\alpha_{t,i}=\frac{\exp e_{t,i}}{\sum_j\exp e_{t,j}},\qquad c_t=\sum_i\alpha_{t,i}h_i\]

Here \(h_i\) is the encoder state at source position \(i\), \(s_{t-1}\) the previous decoder state, and \(c_t\) the context vector for target step \(t\). Weights sum to one over valid source positions. For scores \((0,\log3)\), softmax gives \((0.25,0.75)\); scalar encoder values \((2,6)\) then produce context 5. In the basic model, translation loss trains the scoring network without separate alignment labels. Correspondence-like patterns can be inspected, but attention weights alone do not establish causal explanations of predictions.

import torch
import torch.nn as nn
import torch.nn.functional as F

class AdditiveAttention(nn.Module):
    """Bahdanau attention: score = v^T tanh(W_s s + W_h h)."""
    def __init__(self, dec_dim, enc_dim, attn_dim=128):
        super().__init__()
        self.Ws = nn.Linear(dec_dim, attn_dim, bias=False)
        self.Wh = nn.Linear(enc_dim, attn_dim, bias=False)
        self.v = nn.Linear(attn_dim, 1, bias=False)

    def forward(self, s_prev, enc_states, mask=None):
        # s_prev: (N, dec_dim); enc_states: (N, T, enc_dim)
        if s_prev.ndim != 2 or enc_states.ndim != 3:
            raise ValueError("expected state (N,D) and encoder states (N,T,H)")
        if s_prev.shape[0] != enc_states.shape[0] or enc_states.shape[1] == 0:
            raise ValueError("batch mismatch or empty source")
        if mask is not None:
            if mask.dtype != torch.bool or mask.shape != enc_states.shape[:2]:
                raise ValueError("mask must be boolean (N,T), True for valid positions")
            if not mask.any(dim=1).all():
                raise ValueError("every source must contain a valid position")
        scores = self.v(torch.tanh(
            self.Ws(s_prev).unsqueeze(1) + self.Wh(enc_states))).squeeze(-1)
        if mask is not None:
            scores = scores.masked_fill(~mask, float("-inf"))
        alpha = F.softmax(scores, dim=1)                  # (N, T)
        context = torch.bmm(alpha.unsqueeze(1), enc_states).squeeze(1)
        return context, alpha

torch.manual_seed(0)
attn = AdditiveAttention(256, 512)
mask = torch.tensor([[True]*9, [True]*6 + [False]*3])
ctx, weights = attn(torch.randn(2, 256), torch.randn(2, 9, 512), mask)
print(tuple(ctx.shape), tuple(weights.shape))
# (2, 512) (2, 9)
print(torch.allclose(weights.sum(1), torch.ones(2)), bool((weights[1, 6:] == 0).all()))
# True True

Mask padding before softmax so it gets zero weight. Each example needs at least one valid source position; otherwise all scores are negative infinity and softmax is undefined. The check rejects that case. Encoder states must also be finite: masking scores does not make NaNs in the value vectors safe. Missing masks can change predictions and attention patterns, not just produce an invisible quality loss.

Additive attention projects decoder and encoder states to a common width, applies tanh, and scores with a learned vector. Luong-style scores include a dot product \(s^Th\), requiring equal widths, and a learned bilinear form \(s^TWh\). Transformer attention uses projected queries, keys, and values with scaled scores \(QK^T/\sqrt{d_k}\), usually in multiple heads. It shares the weighted-sum mechanism but is not identical to using raw recurrent states in a dot product.

Dense cross-attention evaluates \(T_{src}T_{tgt}\) source–target pairs, with additional factors for feature widths and scoring operations. This is quadratic when both lengths grow together, and linear in either length when the other is fixed. Keeping all encoder states avoids relying only on the final state, at additional storage cost. It does not remove every memory or optimization limitation.

Teacher forcing and exposure bias

Teacher forcing supplies the true previous target token during training. For target [A, B, EOS], decoder inputs are [BOS, A, B], aligned with prediction targets [A, B, EOS]. Cross-entropy on these predictions is autoregressive maximum-likelihood training. Known inputs avoid feeding sampled mistakes into later training steps, but an RNN still computes states sequentially. A causally masked Transformer can process all these training positions in parallel.

At inference, prefixes contain generated tokens. Their distribution can differ from the reference prefixes used in training; this is called exposure bias. It can affect robustness, but does not by itself explain every repetition or generation error. Scheduled sampling sometimes replaces reference inputs with generated ones. This changes the ordinary likelihood objective and has theoretical consistency concerns, so its benefit needs evaluation rather than assuming it resolves the mismatch.

Connecting attention to a decoder

This synthetic batch shows a complete training step. Token IDs 0, 1, and 2 denote PAD, BOS, and EOS. A packed GRU excludes source padding, its final state initializes the decoder, and each decoder step attends to the encoder outputs. Concatenating the context with the token embedding supplies the GRUCell input; concatenating context with the new state supplies the vocabulary head. This is a compact additive-attention model, not an exact reproduction of the original paper’s decoder.

torch.manual_seed(0)
PAD, BOS, EOS = 0, 1, 2
src = torch.tensor([[3, 4, 5], [4, 5, 0]])
lengths = torch.tensor([3, 2])
target = torch.tensor([[4, 3, EOS], [5, EOS, PAD]])
dec_input = torch.tensor([[BOS, 4, 3], [BOS, 5, EOS]])
source_emb, target_emb = nn.Embedding(6, 4, padding_idx=PAD), nn.Embedding(6, 4, padding_idx=PAD)
encoder = nn.GRU(4, 5, batch_first=True)
decoder = nn.GRUCell(4 + 5, 5)
attention = AdditiveAttention(5, 5, attn_dim=4)
head = nn.Linear(5 + 5, 6)
modules = [source_emb, target_emb, encoder, decoder, attention, head]
parameters = [p for module in modules for p in module.parameters()]
optimizer = torch.optim.SGD(parameters, lr=0.05)
packed = nn.utils.rnn.pack_padded_sequence(source_emb(src), lengths, batch_first=True, enforce_sorted=False)
encoded, last = encoder(packed)
enc_states, _ = nn.utils.rnn.pad_packed_sequence(encoded, batch_first=True, total_length=src.shape[1])
state = last[-1]
logits = []
for t in range(dec_input.shape[1]):
    context, weights = attention(state, enc_states, src != PAD)
    state = decoder(torch.cat([target_emb(dec_input[:, t]), context], dim=-1), state)
    logits.append(head(torch.cat([state, context], dim=-1)))
logits = torch.stack(logits, dim=1)
loss = F.cross_entropy(logits.reshape(-1, 6), target.reshape(-1), ignore_index=PAD)
optimizer.zero_grad()
loss.backward()
print(tuple(logits.shape), int((target != PAD).sum()))
# (2, 3, 6) 5
print(all(p.grad is not None and bool(torch.isfinite(p.grad).all()) for p in parameters))
# True
optimizer.step()

The logits have axes (batch, target time, vocabulary). Cross-entropy averages five valid targets, including both EOS tokens. The second sequence has an unused post-EOS step whose target is padding; it contributes no loss. During generation, replace the true previous token with a selected model token and stop that hypothesis at EOS. Packing handles source-state updates, the attention mask handles source weights, and the loss mask handles target padding: these are separate operations. One successful update checks the wiring, not translation quality.

Greedy search and beam search

Greedy decoding takes the highest-probability token at each step. It is fast and myopic: a locally attractive token can foreclose a much better continuation, and greedy decoding cannot recover.

Beam search maintains a limited set of prefixes, expands them, and prunes candidates by accumulated log-probability. It is approximate because discarded prefixes cannot be recovered. The helper below keeps up to \(B\) active prefixes separately from up to \(B\) completed sequences. It ranks completed sequences by a length-adjusted score. This is one explicit beam policy, not a guarantee of finding the globally best sequence.

import math

def beam_search(step_fn, start, eos, beam_size=3, max_len=20, alpha=0.7):
    """Full prefix -> (token, log probability) list; BOS excluded from length.
    Returns (sequence, raw_log_probability, ended_with_eos).
    """
    if beam_size < 1 or max_len < 1 or alpha < 0 or start == eos:
        raise ValueError("invalid search settings")
    beams = [(0.0, [start])]
    finished = []
    def rank(item):
        score, seq = item
        return score / ((len(seq) - 1) ** alpha)
    for _ in range(max_len):
        candidates = []
        for score, seq in beams:
            for token, logp in step_fn(seq):
                if math.isnan(logp) or logp > 0:
                    raise ValueError("expected log probabilities <= 0")
                if logp == -math.inf:
                    continue
                item = (score + logp, seq + [token])
                if token == eos:
                    finished.append(item)
                else:
                    candidates.append(item)
        finished = sorted(finished, key=rank, reverse=True)[:beam_size]
        beams = sorted(candidates, key=lambda item: item[0], reverse=True)[:beam_size]
        if not beams:
            break
    if finished:
        score, seq = max(finished, key=rank)
        return seq, score, True
    if beams:
        score, seq = max(beams, key=rank)
        return seq, score, False
    raise ValueError("no finite-probability hypothesis")

The helper prefers a completed sequence whenever one exists; only if none finishes does it return a length-capped prefix, flagged as incomplete. It never expands EOS. A real step function must return log probabilities from the conditional model and handle decoder state consistently for every prefix. This simple interface can recompute the prefix; efficient implementations cache and reorder each beam’s state. Tune beam width against quality, latency, and the chosen scoring rule.

Length normalization

Extending a fixed prefix adds a nonpositive log-probability, so its raw score cannot increase. This does not mean every short completed sequence beats every longer one: their prefixes and EOS probabilities differ. Some translation models favor overly short outputs under probability-based search. Length normalization changes the decoding objective and can alter that preference:

\[\text{score}=\frac{1}{L^{\alpha}}\sum_{t=1}^{L}\log P(y_t\mid y_{<t},x)\]

Here \(L\) counts generated tokens including EOS, but excluding BOS. With \(\alpha=0\) this is the raw log-probability; with \(\alpha=1\) it is a mean token log-probability. Choose \(\alpha\) on validation data. The helper’s active prefixes have equal lengths at each expansion, so normalizing them by this length alone would not change that step’s ordering. Comparing completed and active candidates at different lengths, or using early stopping, requires a consistent policy. Final normalization cannot restore a prefix already pruned. The exercise also examines GNMT’s different divisor, \(((5+L)/6)^\alpha\).

BLEU and its limits

BLEU measures n-gram overlap with one or more references. It combines modified n-gram precisions — clipped to reference counts so extra copies cannot create extra matches — through a geometric mean, and multiplies by a brevity penalty that punishes outputs shorter than the reference.

\[\text{BLEU}=BP\cdot\exp\left(\sum_{n=1}^{4}w_n\log p_n\right)\]

An n-gram is a consecutive group of n tokens. If a candidate is “the the cat” and the reference is “the cat”, unigram clipping counts one “the” and one “cat”: precision is 2/3, not 3/3. Only one of the two candidate bigrams matches, giving 1/2. This is a count illustration, not a full four-order BLEU score. Higher-order precisions and the brevity penalty also enter the calculation.

For corpus BLEU, sum clipped match counts and candidate counts across sentences before forming each \(p_n\); do not average sentence BLEU scores. Usually \(w_n=1/4\). The brevity penalty is 1 when candidate length \(c\) exceeds effective reference length \(r\), otherwise \(\exp(1-r/c)\) for \(c>0\); empty output scores zero. With multiple references, effective lengths follow a specified closest-reference rule. Sentence-level variants are noisy and often need smoothing, especially when a precision is zero or high-order n-grams are absent. Correct paraphrases may have low overlap. Compare scores only with matching data and settings, and report a sacreBLEU signature or equivalent tokenization, casing, reference, and smoothing details.

COMET learns an evaluation model from human judgments; BERTScore compares contextual token representations. Such metrics can complement lexical overlap, but their agreement with people varies by task, language, model version, and evaluation data. Report the exact configuration and use human review when the application requires judgments these metrics do not capture.

Self-attention, masking, and the full block are derived in Transformers from Scratch: Self-Attention to Encoder–Decoder.

Implementation checks

• Mask source padding in attention and target padding in the loss. Reject all-masked sources and handle batches with no valid targets explicitly.
• Confirm attention weights sum to 1 across the source axis, not the batch axis.
• Verify that the decoder input is shifted right by one relative to the target.
• Enforce a maximum generation length. Generation may otherwise never terminate if the model does not emit EOS.
• Inspect attention heatmaps alongside predictions and loss. Persistent concentration at position 0 merits investigation, but can reflect a summary token or useful state; the pattern alone does not prove collapse.

How text becomes the integers a model actually sees is covered in Tokenization Explained: BPE, WordPiece, and SentencePiece.

Exercises

1. Searching beyond the greedy prefix. Decode the same toy model greedily and with beam width 4, on a case where the highest-probability first token leads to a worse complete sequence. Report both sequence log-probabilities.

You should get: a beam sequence whose total log-probability beats greedy, reached through a worse first step.

import math
p1 = {"A": 0.6, "B": 0.4}
p2 = {"A": {"x": 0.5, "y": 0.5}, "B": {"x": 0.95, "y": 0.05}}
def step(seq):
    probs = p1 if len(seq) == 1 else p2[seq[1]] if len(seq) == 2 else {"EOS": 1.0}
    return [(token, math.log(prob)) for token, prob in probs.items()]
seq, score = ["BOS"], 0.0
while seq[-1] != "EOS":
    token, logp = max(step(seq), key=lambda item: item[1])
    seq.append(token)
    score += logp
print("greedy", seq[1:-1], round(score, 4))
# greedy ['A', 'x'] -1.204
best, score, complete = beam_search(step, "BOS", "EOS", beam_size=4, max_len=3, alpha=0)
print("beam", best[1:-1], round(score, 4), complete)
# beam ['B', 'x'] -0.9676 True

2. Length normalization changes the answer. Score a short and a long candidate by raw log-probability and by the GNMT length penalty at \(\alpha = 0\), \(0.6\), and \(1.0\). Report which candidate wins under each.

You should get: the ranking reversing between the two ends of the alpha range.

cands = {"short": (-2.5, 4), "long": (-4.0, 12)}   # (log-prob, length)

def lp(length, alpha):
    return ((5 + length) / 6) ** alpha

for alpha in (0.0, 0.6, 1.0):
    scored = {k: v[0] / lp(v[1], alpha) for k, v in cands.items()}
    print(alpha, {k: round(x, 4) for k, x in scored.items()},
          "->", max(scored, key=scored.get))
# 0.0 {'short': -2.5, 'long': -4.0} -> short
# 0.6 {'short': -1.9601, 'long': -2.1413} -> short
# 1.0 {'short': -1.6667, 'long': -1.4118} -> long

3. Token accuracy and exact-match accuracy. Under a toy model with independent token-correctness events of common probability p, calculate the probability that all 20 tokens are correct. Explain why this is not a measurement of exposure bias.

You should get: 0.95 per-token accuracy and about 0.3585 sequence exact-match probability under the stated model.

for acc in (0.99, 0.95, 0.90):
    print(acc, "20 tokens ->", round(acc**20, 4))
# 0.99 20 tokens -> 0.8179
# 0.95 20 tokens -> 0.3585
# 0.9  20 tokens -> 0.1216

References

• Papineni, Roukos, Ward, and Zhu (2002). BLEU: A Method for Automatic Evaluation of Machine Translation. ACL.
• Sutskever, Vinyals, and Le (2014). Sequence to Sequence Learning with Neural Networks. NIPS.
• Bahdanau, Cho, and Bengio (2015). Neural Machine Translation by Jointly Learning to Align and Translate. ICLR.
• Luong, Pham, and Manning (2015). Effective Approaches to Attention-Based Neural Machine Translation. EMNLP.
• Bengio, Vinyals, Jaitly, and Shazeer (2015). Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks. NIPS.
• Wu et al. (2016). Google’s Neural Machine Translation System: Bridging the Gap Between Human and Machine Translation. arXiv preprint.
• Post (2018). A Call for Clarity in Reporting BLEU Scores. WMT.
• Rei, Stewart, Farinha, and Lavie (2020). COMET: A Neural Framework for MT Evaluation. EMNLP.
• Huszár (2015). How (not) to Train your Generative Model: Scheduled Sampling, Likelihood, Adversary?