Sequence models process ordered data such as text, speech, time series, DNA, and video. Unlike ordinary fixed-size inputs, sequences may have different lengths, and the meaning of one element often depends on surrounding elements.

Before constructing a sequence model, it is useful to define notation for:

• Input and output sequences
• Sequence positions
• Variable sequence lengths
• Training examples
• Vocabulary entries
• One-hot vectors, token IDs, and embeddings
• Unknown or unseen tokens

Motivating Example: Named-Entity Recognition

Consider the sentence:

Harry Potter and Hermione Granger invented a new spell.

A named-entity recognition system identifies spans corresponding to entities such as:

• People
• Organizations
• Locations
• Dates
• Countries
• Currencies
• Products

In this simplified example, the system only needs to identify words that are part of a person’s name.

The sentence contains nine tokens:

1. Harry
2. Potter
3. and
4. Hermione
5. Granger
6. invented
7. a
8. new
9. spell

The corresponding simplified binary labels are:\[ (1,1,0,1,1,0,0,0,0) \]

where:

• \(1\) means the token belongs to a person’s name.
• \(0\) means it does not.

Sequence-to-Sequence Labeling

The input is a sequence:\[ X = \left( x^{\langle1\rangle}, x^{\langle2\rangle}, \ldots, x^{\langle T_x\rangle} \right) \]

The output is another sequence:\[ Y = \left( y^{\langle1\rangle}, y^{\langle2\rangle}, \ldots, y^{\langle T_y\rangle} \right) \]

In this named-entity example:\[ T_x=9 \]

and:\[ T_y=9 \]

because the model produces one output label for every input token.

The mapping is:\[ x^{\langle t\rangle} \longrightarrow y^{\langle t\rangle} \]

for each position \(t\).

Position Index \(t\)

The superscript:\[ \langle t\rangle \]

identifies a position in the sequence.

For example:\[ x^{\langle1\rangle} = \text{Harry} \]\[ x^{\langle2\rangle} = \text{Potter} \]\[ x^{\langle3\rangle} = \text{and} \]

Similarly:\[ y^{\langle1\rangle}=1 \]\[ y^{\langle2\rangle}=1 \]\[ y^{\langle3\rangle}=0 \]

The index \(t\) is often interpreted as time, particularly for audio or sensor signals. In text, it simply represents token order.

Input and Output Lengths

Use:\[ T_x \]

for the input-sequence length and:\[ T_y \]

for the output-sequence length.

In token labeling:\[ T_x=T_y \]

However, this equality does not hold for every sequence task.

Task	Relationship between lengths
Part-of-speech tagging	Usually \(T_x=T_y\)
Named-entity recognition	Usually \(T_x=T_y\)
Sentiment classification	\(T_y=1\)
Machine translation	Usually \(T_x\neq T_y\)
Text summarization	Usually \(T_x\neq T_y\)
Speech recognition	Audio length and transcript length differ
Sequence generation	Output length is generated dynamically

A sequence architecture must therefore support both equal-length and unequal-length mappings.

Notation for Training Examples

The notation:\[ X^{(i)} \]

represents training example \(i\).

Its token at sequence position \(t\) is:\[ x^{(i)\langle t\rangle} \]

Its output at that position is:\[ y^{(i)\langle t\rangle} \]

The input length of training example \(i\) is:\[ T_x^{(i)} \]

and its output length is:\[ T_y^{(i)} \]

Therefore:\[ X^{(i)} = \left( x^{(i)\langle1\rangle}, \ldots, x^{(i)\langle T_x^{(i)}\rangle} \right) \]

and:\[ Y^{(i)} = \left( y^{(i)\langle1\rangle}, \ldots, y^{(i)\langle T_y^{(i)}\rangle} \right) \]

Different examples can have different lengths:\[ T_x^{(1)}\neq T_x^{(2)} \]

For example, one sentence may contain nine tokens while another contains fifteen.

Do Not Confuse the Indices

The notation contains two separate indices:

• \((i)\): training-example index
• \(\langle t\rangle\): position within that example

Thus:\[ x^{(7)\langle3\rangle} \]

means the third token in the seventh training example.

Better Labels for Named Entities

The simplified binary representation only indicates whether a token belongs to a person’s name. It does not clearly encode where an entity begins or ends.

For example:

Harry Potter and Hermione Granger

has two separate person entities. Binary labels are:\[ (1,1,0,1,1) \]

The sequence indicates which tokens are names, but a richer labeling scheme makes the boundaries explicit.

BIO Tagging

A common representation is BIO:

• B-PER: beginning of a person entity
• I-PER: inside a person entity
• O: outside an entity

The sentence becomes:

Token	Tag
Harry	B-PER
Potter	I-PER
and	O
Hermione	B-PER
Granger	I-PER
invented	O
a	O
new	O
spell	O

BIO tagging distinguishes adjacent entities and provides explicit span boundaries.

Multiple Entity Types

For a system recognizing several entity categories, the tag set could include:

• B-PER, I-PER
• B-ORG, I-ORG
• B-LOC, I-LOC
• B-DATE, I-DATE
• O

Other schemes such as BIOES add tags for single-token entities and explicit entity endings.

Constructing a Vocabulary

A word-level system begins with a vocabulary:\[ \mathcal{V} = \{ w_1,w_2,\ldots,w_{|\mathcal{V}|} \} \]

Each vocabulary item receives an integer index:\[ \operatorname{index}(w_k)=k \]

For illustration:

Word	Vocabulary index
a	1
Aaron	2
and	367
Harry	4075
Potter	6830
Zulu	10000

The exact indices are arbitrary. What matters is that the mapping remains consistent across training and inference.

A vocabulary with 10,000 words gives:\[ |\mathcal{V}|=10{,}000 \]

Special Tokens

A practical vocabulary often reserves special entries such as:

• <UNK>: unknown token
• <PAD>: padding
• <BOS>: beginning of sequence
• <EOS>: end of sequence
• <MASK>: masked position

Not every model needs every special token.

One-Hot Word Representation

A vocabulary item \(w_k\) can be represented by a one-hot vector:\[ x \in \{0,1\}^{|\mathcal{V}|} \]

with:\[ x_k=1 \]

and:\[ x_j=0 \quad \text{for }j\neq k \]

For a 10,000-word vocabulary, every one-hot vector has 10,000 components.

Example: Harry

If:\[ \operatorname{index}(\text{Harry})=4075 \]

then:\[ x^{\langle1\rangle} = \begin{bmatrix} 0\\ \vdots\\ 0\\ 1\\ 0\\ \vdots\\ 0 \end{bmatrix} \in \mathbb{R}^{10000} \]

where the only nonzero entry is at position 4075.

Example: Potter

If:\[ \operatorname{index}(\text{Potter})=6830 \]

then \(x^{\langle2\rangle}\) has a one at position 6830 and zeros elsewhere.

Example: a

If:\[ \operatorname{index}(\text{a})=1 \]

then:\[ x^{\langle7\rangle} = \begin{bmatrix} 1\\ 0\\ \vdots\\ 0 \end{bmatrix} \]

Limitations of One-Hot Vectors

High Dimensionality

A vocabulary of size \(V\) requires vectors with \(V\) components.

For:\[ V=100{,}000 \]

each vector has 100,000 dimensions, even though only one entry is nonzero.

No Semantic Similarity

Two different one-hot vectors are orthogonal:\[ x_i^Tx_j=0 \quad \text{when }i\neq j \]

Therefore, the representation itself does not indicate that:

• king is related to queen
• cat is related to dog
• Paris is related to France

Out-of-Vocabulary Words

Any word excluded from the vocabulary must be replaced, decomposed, or handled by another mechanism.

Vocabulary Growth

Word-level vocabularies can become extremely large because of:

• Inflections
• Compound words
• Misspellings
• Names
• Domain-specific terminology
• Different scripts and languages

These limitations motivate token IDs, embedding layers, and subword tokenization.

Token IDs and Embedding Lookup

In practical implementations, one-hot vectors are rarely constructed explicitly.

Instead, a token is represented by its integer ID:\[ q^{\langle t\rangle} = \operatorname{index} \left( x^{\langle t\rangle} \right) \]

For example:\[ q^{\langle1\rangle}=4075 \]

An embedding matrix is:\[ E \in \mathbb{R}^{d_e\times|\mathcal{V}|} \]

The embedding for token \(q^{\langle t\rangle}\) is:\[ e^{\langle t\rangle} = E_{:,q^{\langle t\rangle}} \]

If a one-hot vector were used explicitly, the same operation would be:\[ e^{\langle t\rangle} = Ex^{\langle t\rangle} \]

Because \(x^{\langle t\rangle}\) contains only one nonzero value, this multiplication simply selects one column from \(E\).

Embedding Sequence

A sentence becomes:\[ \left( e^{\langle1\rangle}, e^{\langle2\rangle}, \ldots, e^{\langle T_x\rangle} \right) \]

where:\[ e^{\langle t\rangle} \in \mathbb{R}^{d_e} \]

Usually:\[ d_e\ll|\mathcal{V}| \]

For example:\[ |\mathcal{V}|=50{,}000 \]

and:\[ d_e=300 \]

The model processes compact dense vectors rather than extremely large one-hot vectors.

Unknown Words

In a fixed word-level vocabulary, a token not found in the vocabulary is mapped to:\[ \texttt{<UNK>} \]

If the word “spellcrafting” is absent:\[ \operatorname{tokenize} (\text{spellcrafting}) = \texttt{<UNK>} \]

All unknown words then share the same representation.

This prevents lookup failure but loses information about the original word.

Subword Tokenization

Modern language systems commonly use subword units rather than complete words.

An unfamiliar word may be divided into known pieces. For example, a tokenizer might represent a rare word as several reusable fragments rather than replacing the entire word with <UNK>.

Subword tokenization offers:

• Better handling of rare words
• Smaller vocabularies
• Improved representation of names and morphology
• Fewer unknown tokens
• Support for multiple languages and scripts

The sequence positions then refer to subword tokens rather than necessarily corresponding to human-readable words.

Alignment Problem in Token Labeling

Named-entity labels may be provided at the word level, while the model processes subword tokens.

Suppose one word is divided into three subwords. The labels must be aligned using a consistent rule, such as:

• Assign the label only to the first subword.
• Copy the label to every subword.
• Use beginning and continuation labels.
• Ignore continuation subwords in the loss.

This alignment must also be reversed when producing word-level entity spans.

Variable-Length Sequences in a Batch

Neural network batches usually require rectangular tensors, but sentences have different lengths.

Suppose a batch contains lengths:\[ (9,15,11) \]

The shorter examples can be padded to length 15 using <PAD>.

The token-ID tensor then has shape:\[ B\times T_{\max} \]

where:\[ T_{\max} = \max_i T_x^{(i)} \]

Padding Mask

A mask distinguishes real tokens from padding:\[ M^{(i)\langle t\rangle} = \begin{cases} 1, & t\leq T_x^{(i)}\\ 0, & t>T_x^{(i)} \end{cases} \]

The loss should ignore padded positions:\[ J = \frac{ \sum_{i,t} M^{(i)\langle t\rangle} \mathcal{L} \left( y^{(i)\langle t\rangle}, \hat{y}^{(i)\langle t\rangle} \right) }{ \sum_{i,t} M^{(i)\langle t\rangle} } \]

Without masking, the model may be trained to predict labels for artificial padding tokens.

Output Representation

For binary token classification:\[ \hat{y}^{\langle t\rangle} = P \left( \text{person token at position }t \mid X \right) \]

A sigmoid output can be used:\[ \hat{y}^{\langle t\rangle} = \sigma(z^{\langle t\rangle}) \]

For multiclass BIO tagging with \(K\) possible tags:\[ \hat{y}^{\langle t\rangle} \in \mathbb{R}^{K} \]

and:\[ \hat{y}_k^{\langle t\rangle} = \frac{ e^{z_k^{\langle t\rangle}} }{ \sum_{r=1}^{K}e^{z_r^{\langle t\rangle}} } \]

The predicted tag is:\[ \underset{k}{\operatorname{argmax}} \; \hat{y}_k^{\langle t\rangle} \]

Context Matters

The correct label for one word often depends on nearby words.

For example, “Potter” may refer to:

• A person’s surname
• An occupation
• Part of a title
• A fictional character

A sequence model should therefore predict:\[ y^{\langle t\rangle} \]

using contextual information rather than only the isolated token:\[ P \left( y^{\langle t\rangle} \mid x^{\langle1\rangle}, \ldots, x^{\langle T_x\rangle} \right) \]

Depending on the architecture, the relevant context may include:

• Previous tokens
• Future tokens
• The complete sequence
• Learned positional information

Common Sequence Task Patterns

Sequence models support several input–output structures.

Many-to-Many with Equal Lengths

\[ T_x=T_y \]

Examples:

• Named-entity recognition
• Part-of-speech tagging
• Per-frame labeling
• Some anomaly-detection tasks

Many-to-One

\[ T_y=1 \]

Examples:

• Sentiment classification
• Document classification
• Sequence-level diagnosis

One-to-Many

One input produces a sequence.

Examples:

• Conditional text generation
• Image caption generation
• Music generation from a prompt

Many-to-Many with Different Lengths

\[ T_x\neq T_y \]

Examples:

• Translation
• Summarization
• Speech recognition
• Sequence transduction

Common Mistakes

Confusing Example and Sequence Indices

Use:\[ (i) \]

for the training example and:\[ \langle t\rangle \]

for sequence position.

Assuming \(T_x=T_y\) for Every Task

This is true for token labeling but not for translation, summarization, or classification.

Treating a Token ID as a Numerical Quantity

Token ID 6830 is not semantically “larger” than token ID 367. IDs are categorical lookup indices.

Building Dense One-Hot Tensors Unnecessarily

Integer token IDs plus an embedding lookup are more efficient.

Failing to Mask Padding

Padded positions should not contribute to sequence losses or attention calculations.

Using Word-Level Labels Directly on Subwords

Label alignment is required when one word becomes several tokens.

Treating <UNK> as a Complete Solution

Mapping many distinct words to one unknown token discards useful information. Subword tokenization often handles rare words more effectively.

Ignoring Entity Boundaries

Binary labels indicate entity membership but do not always represent separate spans clearly. BIO-style labels are generally more suitable.

Key Takeaway

A sequence is represented as:\[ X = \left( x^{\langle1\rangle}, \ldots, x^{\langle T_x\rangle} \right) \]

with corresponding output:\[ Y = \left( y^{\langle1\rangle}, \ldots, y^{\langle T_y\rangle} \right) \]

For training example \(i\), the token at position \(t\) is:\[ x^{(i)\langle t\rangle} \]

Words or subwords are mapped to vocabulary IDs and then converted into dense embeddings. Sequence lengths may differ across examples, so batching requires padding and masks.

For named-entity recognition, the model produces a label at every token position, preferably using a boundary-aware scheme such as BIO. This notation and representation form the foundation for constructing recurrent, convolutional, attention-based, and other sequence models.