End-to-End Speech Recognition with Attention and CTC

Automatic speech recognition converts an audio signal into a text transcript:\[ x\longrightarrow y. \]

The input \(x\) is a long sequence of acoustic measurements, while the output \(y\) is a much shorter sequence of characters, subword tokens, or words.

Two important approaches to this unequal-length sequence problem are:

• Encoder–decoder models with attention
• Connectionist Temporal Classification, or CTC

The Speech Recognition Problem

Suppose an audio recording contains the phrase:

“the quick brown fox”

A speech-recognition system receives a waveform or acoustic feature sequence and should produce the corresponding transcript.

Let\[ x=(x_1,x_2,\ldots,x_{T_x}) \]

be the acoustic sequence and\[ y=(y_1,y_2,\ldots,y_{T_y}) \]

be the transcript.

Usually,\[ T_x\gg T_y. \]

A short utterance may contain hundreds or thousands of acoustic frames but only a few dozen text tokens.

What an Audio Waveform Represents

A microphone converts changes in air pressure into a sampled electrical or digital signal.

The waveform can be represented as\[ s[n], \qquad n=1,\ldots,N, \]

where \(n\) indexes audio samples.

At a sampling rate of 16 kHz, one second of audio contains\[ 16{,}000 \]

samples. Processing every sample as a separate recurrent time step would create extremely long sequences.

Speech systems therefore commonly transform the waveform into a more compact time–frequency representation or use a learned front end that performs temporal downsampling.

Framing the Audio

The waveform is divided into short overlapping frames.

A typical conceptual configuration might use:

• Frame length of roughly 20–30 milliseconds
• Frame shift of roughly 10 milliseconds

With a 10-millisecond shift, the feature rate is approximately\[ 100\text{ frames per second}. \]

A 10-second recording therefore produces roughly\[ T_x=1000 \]

feature frames before additional downsampling.

Spectrogram Features

A short-time Fourier transform analyzes the frequency content of each frame:\[ X(m,k) = \sum_n s[n]\,w[n-mH]e^{-j2\pi kn/N}, \]

where:

• \(w\) is a window function
• \(H\) is the frame hop
• \(m\) indexes time frames
• \(k\) indexes frequency bins

A spectrogram commonly uses\[ |X(m,k)|^2 \]

or its logarithm.

The resulting representation shows:

• Time horizontally
• Frequency vertically
• Energy through intensity

Speech sounds create characteristic time-varying frequency patterns.

Common Acoustic Representations

Speech systems may use:

• Linear-frequency spectrograms
• Log-mel spectrograms
• Mel-frequency cepstral coefficients
• Filterbank energies
• Learned waveform features

A log-mel feature can be written conceptually as\[ x_t = \log \left( M|X_t|^2+\epsilon \right), \]

where \(M\) is a mel filterbank matrix.

Learned front ends can reduce reliance on manually selected features, but they still perform structured transformations and downsampling internally.

Why Time–Frequency Features Help

A raw waveform’s local values depend strongly on phase and sampling alignment. Frequency-domain features expose patterns more closely related to:

• Vowels
• Consonants
• Pitch
• Formants
• Energy transitions
• Frication
• Silence

Human hearing also performs frequency-selective analysis, although biological hearing is much more complex than a conventional spectrogram.

Traditional Intermediate Units

Earlier speech-recognition pipelines often decomposed speech into phonetic units and combined several separately designed components:

• Acoustic models
• Pronunciation lexicons
• Phoneme sequences
• Language models
• Decoding graphs

End-to-end neural systems can instead learn a mapping from acoustic input to text targets with fewer manually specified intermediate representations.

This does not mean that pronunciation or phonetic structure becomes irrelevant. Rather, the model learns useful internal representations without requiring phoneme labels as mandatory supervision.

Data Requirements

Speech varies across:

• Speakers
• Accents
• Speaking rates
• Microphones
• Rooms
• Background noise
• Domains
• Vocabulary
• Emotional states

A robust general-purpose system therefore benefits from large and diverse paired datasets:\[ (\text{audio},\text{transcript}). \]

Smaller datasets can still support narrow-domain recognition, adaptation, or fine-tuning, especially when pretrained speech encoders are available.

Data quality and diversity are often as important as raw duration.

Acoustic Encoder

A modern speech model normally begins with an encoder:\[ h_1,h_2,\ldots,h_{T’} = \operatorname{Encoder}(x_1,\ldots,x_{T_x}), \]

where\[ T’\leq T_x. \]

The encoder may include:

• Convolutional downsampling
• Bidirectional recurrent layers
• Unidirectional recurrent layers
• Attention-based layers
• Normalization
• Residual connections

Downsampling reduces computational cost and removes unnecessary frame-level redundancy.

Attention-Based Speech Recognition

An attention decoder generates text one token at a time.

At output step \(u\), it computes attention over acoustic encoder states:\[ e_{u,t} = \operatorname{score}(s_{u-1},h_t), \]\[ \alpha_{u,t} = \frac{ \exp(e_{u,t}) }{ \sum_j\exp(e_{u,j}) }, \]\[ c_u = \sum_t\alpha_{u,t}h_t. \]

The decoder predicts\[ P(y_u\mid y_{<u},x) = \operatorname{softmax} (W_os_u+U_oc_u+b_o). \]

The output length is independent of the number of acoustic frames.

Advantages of Attention-Based Recognition

An attention decoder can:

• Learn a soft alignment between audio and text
• Model output-token dependencies directly
• Produce variable-length transcripts
• Use rich decoder context
• Integrate acoustic and linguistic information

However, unconstrained attention can struggle with:

• Very long recordings
• Streaming requirements
• Skipped audio regions
• Repeated phrases
• Nonmonotonic alignment errors

Speech is naturally monotonic, so many systems impose or encourage monotonic attention behavior.

Connectionist Temporal Classification

CTC addresses the length mismatch without requiring frame-level alignment labels.

It assumes the encoder produces one distribution at every output frame:\[ P(\pi_t\mid x), \qquad t=1,\ldots,T’. \]

Each frame predicts a symbol from an augmented alphabet containing:

• Ordinary output symbols
• A special blank symbol

Let the blank be denoted by\[ \varnothing. \]

It is distinct from an ordinary space character.

CTC Paths

A CTC path is a frame-level symbol sequence\[ \pi=(\pi_1,\pi_2,\ldots,\pi_{T’}). \]

For example, a path might be:\[ \text{t},\text{t},\varnothing, \text{h},\varnothing, \text{e},\text{e}, \varnothing. \]

CTC maps this long path into a shorter transcript using a collapse function\[ B(\pi). \]

The CTC Collapse Rule

The collapse operation has two ordered steps:

1. Merge consecutive repeated nonblank symbols.
2. Remove blank symbols.

For example:\[ \pi = (\text{t},\text{t},\varnothing, \text{h},\text{h},\varnothing, \text{e},\text{e}) \]

first becomes\[ (\text{t},\varnothing, \text{h},\varnothing, \text{e}) \]

and then\[ B(\pi) = (\text{t},\text{h},\text{e}). \]

The order matters.

Why Blank Is Necessary

The blank symbol serves several purposes:

• It represents frames with no emitted text symbol.
• It separates repeated labels.
• It allows long acoustic sequences to map to short transcripts.
• It avoids requiring exact frame boundaries.

Suppose the desired output contains repeated letters, such as “ll.”

The path\[ (\text{l},\text{l}) \]

collapses to one “l” because the repeats are consecutive.

To produce two “l” characters, a blank must separate them:\[ (\text{l},\varnothing,\text{l}) \longrightarrow (\text{l},\text{l}). \]

This is a central CTC detail.

Blank Is Not a Space

The CTC blank\[ \varnothing \]

means “emit no label at this frame.”

A word-separating space is an ordinary output symbol.

Therefore:\[ \varnothing\neq\text{space}. \]

A character-based transcript such as

“the quick”

requires an actual space token between the words, while blank tokens may appear anywhere in the frame-level path.

Many Paths Can Produce the Same Transcript

The transcript “the” may correspond to paths such as:\[ (\text{t},\text{h},\text{e}), \]\[ (\text{t},\text{t},\text{h},\text{e}), \]\[ (\varnothing,\text{t},\varnothing, \text{h},\text{e},\varnothing), \]

or many others.

All paths satisfying\[ B(\pi)=y \]

represent the same target transcript.

CTC does not force the training data to specify which frame corresponds to each character.

CTC Sequence Probability

The probability of a path is commonly factorized as\[ P(\pi\mid x) = \prod_{t=1}^{T’} P(\pi_t\mid x). \]

The probability of transcript \(y\) is the sum over all compatible paths:\[ P(y\mid x) = \sum_{\pi:B(\pi)=y} P(\pi\mid x). \]

The CTC loss is\[ \mathcal{L}_{\text{CTC}} = -\log P(y\mid x). \]

This summation is essential. Training does not choose one arbitrary alignment and treat it as the truth.

Dynamic Programming

The number of valid paths can be enormous, so they cannot be enumerated directly.

CTC uses a forward–backward dynamic program to compute\[ P(y\mid x) \]

efficiently.

The transcript is expanded by inserting blanks:\[ y’ = (\varnothing,y_1,\varnothing,y_2, \ldots,\varnothing,y_U,\varnothing). \]

The dynamic program tracks how much probability mass reaches each position in this expanded sequence at each acoustic frame.

This allows gradients to be calculated without explicit frame-level alignment labels.

CTC Length Requirement

The encoder output must be long enough to represent the target.

At minimum,\[ T’\geq U, \]

where \(U\) is target length.

If the target contains adjacent repeated symbols, additional frames are needed because blanks must separate them.

For example, representing “ll” requires at least:\[ (\text{l},\varnothing,\text{l}), \]

so at least three CTC time steps are necessary.

Excessive temporal downsampling can therefore make some targets impossible under CTC.

CTC Conditional Independence Assumption

In the standard CTC factorization, frame predictions are conditionally independent given the encoded input:\[ P(\pi\mid x) = \prod_tP(\pi_t\mid x). \]

The encoder can use broad acoustic context, especially when bidirectional, but the output probability at one frame does not explicitly condition on previously emitted nonblank labels.

This is weaker than an autoregressive decoder:\[ P(y_u\mid y_{<u},x). \]

An external language model or a more expressive transducer can compensate for this limitation.

Decoding CTC Outputs

Greedy CTC decoding

At each frame, choose\[ \pi_t^* = \arg\max_kP(k\mid x). \]

Then collapse:\[ \hat{y}=B(\pi^*). \]

This is fast but may miss the most probable transcript because the most probable individual path need not correspond to the most probable collapsed sequence.

CTC beam search

Maintain several prefix hypotheses while combining probabilities of different paths that collapse to the same transcript.

A CTC-specific beam search must distinguish between prefixes ending in:

• Blank
• Nonblank

This is necessary for correctly handling repeated symbols.

Adding a Language Model

CTC decoding can combine acoustic and language-model scores:\[ S(y,x) = \log P_{\text{CTC}}(y\mid x) + \lambda\log P_{\text{LM}}(y) + \beta |y|. \]

Here:

• \(P_{\text{CTC}}\) measures acoustic compatibility
• \(P_{\text{LM}}\) measures linguistic plausibility
• \(\lambda\) controls language-model influence
• \(\beta\) is an insertion or length adjustment

These weights are tuned on held-out data.

Attention and CTC Compared

Property	Attention decoder	CTC
Alignment	Learned soft attention	Marginalized monotonic paths
Decoder	Autoregressive	Framewise output model
Output dependencies	Explicit	Limited in standard form
Streaming	Requires suitable attention design	Easier with causal encoder
Training alignment labels	Not required	Not required
Natural alignment assumption	Flexible	Monotonic
Decoding	Autoregressive search	CTC collapse or prefix search

Speech has approximately monotonic alignment, making CTC a natural fit.

Hybrid CTC–Attention Models

A model can use both objectives:\[ \mathcal{L} = \lambda \mathcal{L}_{\text{CTC}} + (1-\lambda) \mathcal{L}_{\text{att}}. \]

CTC can encourage monotonic alignment and stabilize training, while the attention decoder models output dependencies more directly.

During decoding, scores from both components may also be combined.

Streaming Recognition

An offline bidirectional encoder can inspect the complete utterance. A streaming recognizer must operate without unlimited future context.

Possible streaming components include:

• Causal convolutions
• Unidirectional recurrent layers
• Limited right context
• Chunked processing
• Monotonic attention
• Transducer architectures

The design must trade off:

• Accuracy
• Latency
• Memory
• Computational cost
• Stability of partial transcripts

Output Units

Speech recognition can predict different types of text units.

Characters

Advantages:

• Small vocabulary
• Natural handling of unseen words

Disadvantages:

• Long output sequences
• More decoding steps

Subword tokens

Advantages:

• Shorter sequences than characters
• Better rare-word handling than word-only vocabularies
• Practical vocabulary size

Subword units are widely used in modern systems.

Words

Advantages:

• Short output sequences

Disadvantages:

• Very large vocabulary
• Out-of-vocabulary problems
• Sparse coverage of rare words

Phonemes

Phoneme targets remain useful in some systems, especially when pronunciation modeling, multilingual transfer, or lexicon constraints matter. End-to-end modeling does not make them universally obsolete.

Evaluating Speech Recognition

A standard metric is word error rate:\[ \operatorname{WER} = \frac{S+D+I}{N}, \]

where:

• \(S\) is the number of substitutions
• \(D\) is the number of deletions
• \(I\) is the number of insertions
• \(N\) is the number of reference words

For character-based languages or tasks, character error rate may be used:\[ \operatorname{CER} = \frac{S_c+D_c+I_c}{N_c}. \]

Normalization conventions—such as case folding, punctuation removal, and number formatting—must be consistent.

Common Mistakes

Confusing blank with space

The CTC blank emits nothing. A space is part of the transcript.

Removing blanks before merging repeats

CTC first merges consecutive repeated labels and then removes blanks.

Selecting only one alignment during training

The CTC loss sums over every valid alignment path.

Downsampling too aggressively

The resulting sequence may be too short to represent the target, especially when it contains repeated symbols.

Assuming framewise argmax gives the best transcript

The most probable path and most probable collapsed sequence can differ.

Treating end-to-end as preprocessing-free

Even waveform models use learned filtering, downsampling, normalization, and other architectural structure.

Assuming phonetic units have no value

They are not mandatory, but they remain useful in some settings.

Evaluating without text normalization conventions

Tokenization and normalization can materially change error rates.

Key Takeaway

Speech recognition maps a long acoustic sequence to a shorter text sequence:\[ x_1,\ldots,x_{T_x} \longrightarrow y_1,\ldots,y_{T_y}, \qquad T_x\gg T_y. \]

Attention-based models learn soft alignments and generate transcript tokens autoregressively. CTC instead predicts frame-level symbols from an alphabet augmented with a blank and sums the probabilities of all paths that collapse to the correct transcript:\[ P(y\mid x) = \sum_{\pi:B(\pi)=y} \prod_tP(\pi_t\mid x). \]

CTC resolves the unknown alignment through dynamic programming, while attention provides flexible target-specific access to encoder states. Both approaches avoid requiring manually labeled phoneme boundaries and form important foundations for end-to-end speech recognition.