Sequence-to-Sequence Models and Encoder–Decoder Architectures

Sequence-to-sequence models map an input sequence to an output sequence whose length may be different. They provide a general framework for tasks such as machine translation, speech recognition, summarization, and structured text generation.

The basic architecture contains two components:

An encoder that converts the input into a learned representation
A decoder that generates the output one token at a time

The Sequence-to-Sequence Problem

Suppose the input is the French sentence:

“Jane visite l’Afrique en septembre.”

A possible English translation is:

“Jane visits Africa in September.”

Represent the source as\[ x=(x_1,x_2,\ldots,x_{T_x}) \]

and the target as\[ y=(y_1,y_2,\ldots,y_{T_y}). \]

The lengths do not need to match:\[ T_x\neq T_y. \]

This makes the problem different from token-labeling tasks in which every input position has one corresponding output.

Why a Standard Many-to-Many RNN Is Insufficient

An aligned many-to-many RNN assumes that output \(y_t\) corresponds directly to input \(x_t\). This works for tasks such as part-of-speech tagging.

Translation has no such fixed alignment:

One source word may require several target words.
Several source words may become one target word.
Word order may change.
Some grammatical elements may have no direct counterpart.

The model therefore needs separate input-processing and output-generation stages.

The Encoder

The encoder reads the source sequence one token at a time:\[ h_t^{\text{enc}} = f_{\text{enc}} (h_{t-1}^{\text{enc}},e_x(x_t)), \]

where:

\(e_x(x_t)\) is the source-token embedding
\(h_t^{\text{enc}}\) is the encoder state
\(f_{\text{enc}}\) may be an RNN, GRU, or LSTM

After reading the complete sequence, the simplest model uses the final state as a summary:\[ c=h_{T_x}^{\text{enc}}. \]

The vector \(c\) is commonly called a context vector or sentence representation.

The encoder performs\[ (x_1,\ldots,x_{T_x}) \longrightarrow c. \]

GRU and LSTM Encoders

For a GRU encoder,\[ h_t^{\text{enc}} = \operatorname{GRU}_{\text{enc}} (e_x(x_t),h_{t-1}^{\text{enc}}). \]

For an LSTM encoder,\[ (h_t^{\text{enc}},c_t^{\text{enc}}) = \operatorname{LSTM}_{\text{enc}} (e_x(x_t),h_{t-1}^{\text{enc}},c_{t-1}^{\text{enc}}). \]

In the LSTM case, the decoder may be initialized from both the final hidden state and final cell state.

Gated cells are generally better than a basic RNN at preserving information across longer source sequences.

The Decoder

The decoder generates the target autoregressively:\[ y_1,y_2,\ldots,y_{T_y}. \]

Its initial state is derived from the encoder representation:\[ h_0^{\text{dec}} = g(c). \]

At time \(t\), it receives the preceding target token:\[ h_t^{\text{dec}} = f_{\text{dec}} (h_{t-1}^{\text{dec}},e_y(y_{t-1})). \]

It then predicts a distribution over the target vocabulary:\[ P(y_t\mid y_{<t},x) = \operatorname{softmax} (W_oh_t^{\text{dec}}+b_o). \]

The decoder continues until it produces an end-of-sequence token.

Boundary Tokens

Two special tokens are important.

Beginning-of-sequence token

<BOS> provides the first decoder input:\[ y_0=\text{<BOS>}. \]

End-of-sequence token

<EOS> marks completion:\[ y_{T_y}=\text{<EOS>}. \]

The model learns both the target content and when generation should stop.

Conditional Sequence Probability

The encoder–decoder model estimates\[ P(y\mid x). \]

Using the chain rule,\[ P(y\mid x) = \prod_{t=1}^{T_y} P(y_t\mid y_{<t},x). \]

The source sequence influences the decoder through its encoded representation or, in attention-based models, through the complete set of encoder states.

This makes the decoder a conditional language model.

Training with Teacher Forcing

During training, the decoder usually receives the correct preceding target token.

For the translation

“Jane visits Africa in September.”

the shifted decoder sequence is:

Decoder input	Target
`<BOS>`	Jane
Jane	visits
visits	Africa
Africa	in
in	September
September	`<EOS>`

Thus,\[ \text{decoder input at }t = y_{t-1}^{\text{true}}. \]

This strategy is called teacher forcing.

Translation Loss

For one source–target pair, the negative log-likelihood is\[ \mathcal{L} = -\sum_{t=1}^{T_y} \log P(y_t^{\text{true}} \mid y_{<t}^{\text{true}},x). \]

Equivalently, a cross-entropy loss is computed at every valid decoder position.

For a padded batch, padding positions must be masked:\[ \mathcal{L} = \frac{ \sum_{n,t} m_{n,t}\ell_{n,t} }{ \sum_{n,t}m_{n,t} }. \]

Here, \(m_{n,t}=0\) for padding and 1 for a genuine target token.

Training and Inference Differ

During training, the decoder receives the correct previous token:\[ y_{t-1}^{\text{true}}. \]

During inference, the correct target is unknown. The decoder receives a token selected from its own preceding prediction:\[ \hat{y}_{t-1}. \]

This difference can cause error accumulation. An incorrect early token changes the context for every later prediction.

The phenomenon is often called exposure bias.

Choosing the Output Sequence

The model defines a probability distribution over possible target sequences. A decoding procedure must select an output.

Possible methods include:

Greedy decoding
Beam search
Random sampling
Constrained search

For translation or transcription, the usual goal is a reliable, high-scoring output:\[ y^* = \arg\max_y P(y\mid x). \]

Random sampling is more useful when diversity is desirable.

Greedy Decoding

Greedy decoding chooses the most probable token at every step:\[ \hat{y}_t = \arg\max_w P(w\mid\hat{y}_{<t},x). \]

This is fast but may miss a better complete sequence because a locally optimal token does not necessarily lead to the globally best continuation.

Beam Search

Beam search preserves several promising prefixes at each step.

With beam width \(B\), it:

Expands each of the current \(B\) hypotheses.
Scores their possible next tokens.
Retains the best \(B\) resulting prefixes.
Continues until enough hypotheses reach <EOS>.

Beam search is approximate, but it typically explores the output space more effectively than greedy decoding.

The Fixed-Vector Bottleneck

The original architecture compresses the complete source into one vector:\[ c=h_{T_x}^{\text{enc}}. \]

This can work for short inputs but becomes difficult as source length increases.

The encoder must preserve:

Meaning
Word order
Names
Numbers
Grammatical relationships
Details needed much later by the decoder

A fixed-dimensional bottleneck can lose information.

Attention

Attention lets the decoder consult all encoder states:\[ h_1^{\text{enc}}, h_2^{\text{enc}}, \ldots, h_{T_x}^{\text{enc}}. \]

At decoder step \(t\), it computes weights\[ \alpha_{t,1}, \alpha_{t,2}, \ldots, \alpha_{t,T_x}, \]

satisfying\[ \sum_{i=1}^{T_x}\alpha_{t,i}=1. \]

The decoder-specific context is\[ c_t = \sum_{i=1}^{T_x} \alpha_{t,i}h_i^{\text{enc}}. \]

This allows different output tokens to focus on different parts of the source.

For example:

Generating “Jane” can focus on “Jane.”
Generating “Africa” can focus on “l’Afrique.”
Generating “September” can focus on “septembre.”

Bidirectional Encoders

Because the full source sentence is available before decoding, the encoder can process it in both directions:\[ \overrightarrow{h}_t = f_{\rightarrow} (\overrightarrow{h}_{t-1},e_x(x_t)), \]\[ \overleftarrow{h}_t = f_{\leftarrow} (\overleftarrow{h}_{t+1},e_x(x_t)). \]

The encoder representation at position \(t\) becomes\[ h_t^{\text{enc}} = [ \overrightarrow{h}_t; \overleftarrow{h}_t ]. \]

Each source position then contains information from both earlier and later source tokens.

Image Captioning as Encoder–Decoder Modeling

The same general architecture can map an image to a text sequence.

Image encoder

A convolutional network or vision encoder transforms an image \(I\) into a feature representation:\[ c=f_{\text{vision}}(I). \]

In an older architecture, this may be a single feature vector from a late convolutional or fully connected layer.

Caption decoder

A recurrent decoder generates a caption:\[ P(y\mid I) = \prod_{t=1}^{T_y} P(y_t\mid y_{<t},I). \]

For example:\[ I \longrightarrow \text{“A cat sitting on a chair.”} \]

The image representation replaces the encoded source sentence as the decoder’s conditioning information.

Spatial Attention for Image Captioning

Compressing an image into one vector can discard spatial detail. A stronger approach retains a set of visual features:\[ v_1,v_2,\ldots,v_M, \]

where each vector corresponds to an image region or patch.

At word step \(t\), attention produces\[ c_t = \sum_{i=1}^{M} \alpha_{t,i}v_i. \]

The decoder can focus on:

The cat while generating “cat”
The chair while generating “chair”
A spatial relationship while generating “on”

This mirrors attention over source-language positions in translation.

Training an Image Captioner

Given an image \(I\) and caption\[ y_1,\ldots,y_T, \]

the loss is\[ \mathcal{L} = -\sum_{t=1}^{T} \log P(y_t\mid y_{<t},I). \]

The visual encoder may be:

Frozen
Partially fine-tuned
Fully trained jointly with the decoder

Freezing can help with limited caption data, while fine-tuning can adapt visual features to the target domain.

Other Encoder–Decoder Applications

The framework applies whenever one structured input must produce a variable-length sequence.

Input	Output	Application
Source-language text	Target-language text	Machine translation
Audio signal	Text	Speech recognition
Document	Shorter text	Summarization
Image	Text	Image captioning
Question and context	Answer sequence	Generative question answering
Structured data	Description	Data-to-text generation

The encoder type can change while the decoder remains an autoregressive sequence model.

A Minimal Recurrent Model

A simplified recurrent sequence-to-sequence structure can be written as:

class Seq2SeqModel:
    def encode(self, source_tokens):
        source_vectors = self.source_embedding(source_tokens)
        encoder_outputs, encoder_state = self.encoder(
            source_vectors
        )
        return encoder_outputs, encoder_state

    def decode_step(
        self,
        previous_token,
        decoder_state,
        encoder_outputs
    ):
        token_vector = self.target_embedding(
            previous_token
        )

        decoder_output, next_state = self.decoder(
            token_vector,
            decoder_state
        )

        logits = self.output_layer(decoder_output)
        return logits, next_state

class Seq2SeqModel:
    def encode(self, source_tokens):
        source_vectors = self.source_embedding(source_tokens)
        encoder_outputs, encoder_state = self.encoder(
            source_vectors
        )
        return encoder_outputs, encoder_state

    def decode_step(
        self,
        previous_token,
        decoder_state,
        encoder_outputs
    ):
        token_vector = self.target_embedding(
            previous_token
        )

        decoder_output, next_state = self.decoder(
            token_vector,
            decoder_state
        )

        logits = self.output_layer(decoder_output)
        return logits, next_state

A full implementation must additionally manage:

Padding masks
LSTM cell states
Attention
Batched variable-length sequences
Start and end tokens
Decoding strategy
State reordering during beam search

Common Mistakes

Assuming source and target lengths must match

Sequence-to-sequence models explicitly support\[ T_x\neq T_y. \]

Feeding predicted tokens during all training steps

Teacher forcing normally uses correct preceding targets during supervised training.

Forgetting `<EOS>`

Without an end token or another stopping rule, the decoder cannot learn when to stop.

Using random sampling when one reliable output is required

Sampling is useful for diversity, but translation and transcription generally require high-scoring outputs.

Treating the final encoder state as lossless

One vector can become a serious bottleneck for long or information-rich inputs.

Ignoring padding

Padding positions must not contribute to attention or loss.

Assuming fluent output is faithful

A decoder can produce grammatical text that omits or invents source information.

Key Takeaway

A sequence-to-sequence model uses an encoder to represent an input and a decoder to generate a variable-length output:\[ x \longrightarrow \text{encoder representation} \longrightarrow y. \]

For translation,\[ P(y\mid x) = \prod_{t=1}^{T_y} P(y_t\mid y_{<t},x). \]

The decoder generates tokens autoregressively until it produces an end-of-sequence symbol. The same framework can map images to captions by replacing the text encoder with a visual encoder.

The simplest model compresses the complete input into one vector, while attention-based systems allow the decoder to access different input positions or image regions throughout generation.