Speech Recognition, Synthesis, and Streaming

Consider a hypothetical phone assistant for an order desk. A caller says, “Do not cancel order fifteen.” The system must turn the sound into text, decide what to do, and speak a reply, all while the caller waits in real time. If the recognizer drops the word “not”, the order is canceled. If it hears “fifty”, the wrong order is touched. If the system waits for the caller to stop, thinks for two seconds, and then synthesizes the whole reply before playing anything, the caller hangs up or talks over it. Each of these is a different kind of failure, and each has its own mechanism and its own measurement.

A speech system is a chain of models around a signal-processing front end. Audio is sampled and turned into a time-frequency representation. A recognizer converts that representation into text without knowing in advance which frame belongs to which word. A synthesizer goes the other direction and has to invent timing, pitch, and a waveform that the text never specified. A streaming layer decides when each stage is allowed to commit, because a system that needs the whole utterance before it starts cannot hold a conversation.

Readers should bring the encoder, tokenization, and cross-modal alignment ideas from the preceding article on multimodal models, which treats audio as one input modality among several. It also assumes sequence models: recurrent networks for the idea of state carried along a sequence, and transformers for self-attention and encoder-decoder cross-attention. The glossary entries on speech recognition (ASR) and text-to-speech (TTS) give the short definitions.

From pressure waves to model input

Sampling and frames

The multimodal article builds the log-mel front end step by step: window, FFT, mel filterbank, logarithm. Only the speech-specific consequences are needed here. Speech recognition commonly works at 16 kHz, which keeps content up to 8 kHz (the Nyquist frequency, half the sampling rate) and covers the bands that carry most of the information needed to tell speech sounds apart. Music and high-quality synthesis use 22.05, 24, or 44.1 kHz and above. Content above the Nyquist frequency does not disappear when sampled; it folds back into lower frequencies, as the glossary entry on signal processing demonstrates with numbers. Resampling from one rate to another therefore needs a low-pass filter first.

The front end cuts the waveform into overlapping frames, because the sound of a vowel or a consonant is roughly stable over 20 to 30 ms but changes over 100 ms. For \(N\) samples, a frame of \(n\) samples, and a hop of \(h\) samples, the number of frames is

\[ T = 1 + \left\lfloor \frac{N – n}{h} \right\rfloor . \]

With \(n = 400\) (25 ms at 16 kHz) and \(h = 160\) (10 ms), one second of audio gives 98 frames and thirty seconds gives 2,998, about 100 frames per second. The frame length \(n\) also fixes the frequency resolution, which is the sampling rate divided by \(n\): 40 Hz per FFT bin for 400 samples. Frame length is therefore a trade between resolving frequency and resolving time, and the model sees whatever resolution the front end gives it. Whisper, described below, takes 80 log-mel channels with exactly the 25 ms window and 10 ms hop used here.

The following program, run with Python 3.12 and NumPy 2.5, measures that trade-off and the resampling hazard on synthetic signals. It analyzes two steady tones 100 Hz apart with windows of 10, 25, and 100 ms; a 5 ms click with the same three windows and a 10 ms hop; and a 6 kHz tone decimated to 8 kHz with and without a low-pass filter.

import numpy as np

SR = 16000

def n_frames(n_samples, n_fft, hop):
    return 1 + (n_samples - n_fft) // hop

def stft_power(x, n_fft, hop):
    win = np.hanning(n_fft)
    T = n_frames(len(x), n_fft, hop)
    frames = np.stack([x[t * hop: t * hop + n_fft] * win for t in range(T)])
    return np.abs(np.fft.rfft(frames, axis=1)) ** 2

print("frames for 1 s and 30 s at 25 ms / 10 ms:", n_frames(SR, 400, 160), n_frames(30 * SR, 400, 160))

t = np.arange(2 * SR) / SR

# 1. Frequency resolution: two steady tones 100 Hz apart, three window lengths.
two = np.sin(2 * np.pi * 1000 * t) + np.sin(2 * np.pi * 1100 * t)
print("\ntwo tones at 1000 and 1100 Hz")
print("%10s %14s %24s" % ("window", "bin spacing", "spectral peaks near 1 kHz"))
for ms in (10, 25, 100):
    n = int(SR * ms / 1000)
    sp = stft_power(two, n, n // 2)[5]
    peaks = [i for i in range(1, len(sp) - 1) if sp[i] > sp[i - 1] and sp[i] > sp[i + 1] and sp[i] > 0.05 * sp.max()]
    print("%7d ms %11.0f Hz %24d" % (ms, SR / n, len(peaks)))

# 2. Time resolution: a 5 ms click; how many 10 ms-hop frames carry its energy?
click = np.zeros(2 * SR)
click[SR: SR + 80] = np.hanning(80)
print("\na 5 ms click, hop 10 ms")
print("%10s %28s" % ("window", "frames above 10% of the peak"))
for ms in (10, 25, 100):
    n = int(SR * ms / 1000)
    e = stft_power(click, n, 160).sum(1)
    print("%7d ms %28d" % (ms, int((e > 0.1 * e.max()).sum())))

# 3. Decimating 16 kHz -> 8 kHz by keeping every second sample, with and without a low-pass.
x = np.sin(2 * np.pi * 6000 * t)           # 6 kHz is above the new Nyquist frequency of 4 kHz
naive = x[::2]
spec = np.abs(np.fft.rfft(naive))
print("\nnaive decimation of a 6 kHz tone: peak at %.0f Hz, amplitude %.2f"
      % (np.fft.rfftfreq(len(naive), 1 / 8000)[spec.argmax()], np.abs(naive).max()))
lp = np.convolve(x, np.sinc(np.arange(-31, 32) * 0.5) * 0.5 * np.hamming(63), mode="same")[::2]
print("low-pass (windowed sinc, cutoff 4 kHz) before decimating: amplitude %.2f" % np.abs(lp).max())

frames for 1 s and 30 s at 25 ms / 10 ms: 98 2998

two tones at 1000 and 1100 Hz
    window    bin spacing spectral peaks near 1 kHz
     10 ms         100 Hz                        1
     25 ms          40 Hz                        2
    100 ms          10 Hz                        2

a 5 ms click, hop 10 ms
    window frames above 10% of the peak
     10 ms                            1
     25 ms                            1
    100 ms                            6

naive decimation of a 6 kHz tone: peak at 2000 Hz, amplitude 1.00
low-pass (windowed sinc, cutoff 4 kHz) before decimating: amplitude 0.14

The first line confirms the frame arithmetic. With a 10 ms window the bins are 100 Hz apart and the Hann window’s main lobe is several bins wide, so the two tones merge into one peak. At 25 ms the bins are 40 Hz apart and two peaks appear, and at 100 ms they are well separated. The click shows the opposite pull: with a 100 ms window its energy spreads across 6 frames at a 10 ms hop, so the position of the onset is blurred, while the shorter windows keep it in one frame. A recognizer must tell vowels (long, spectrally distinct) from stop consonants (short bursts), and the 25 ms window with a 10 ms hop is a compromise between those needs, not a law.

The last two lines are the resampling trap. Keeping every second sample of a 16 kHz signal gives an 8 kHz signal whose Nyquist frequency is 4 kHz. The 6 kHz tone does not vanish; it reappears at 2 kHz with full amplitude. With a windowed-sinc low-pass filter applied first, the amplitude drops to 0.14 (a longer filter would attenuate more). A training set that mixes 8 kHz telephone audio with 48 kHz studio audio needs this step done correctly in the resampler, or the data contains frequencies that were not in the original sound.

Learned front ends

The log-mel spectrogram is designed by hand, and learned front ends replace the design with training. wav2vec 2.0, by Baevski and colleagues, feeds the raw waveform to a stack of convolutions, masks part of the resulting latent sequence, and trains a transformer to identify the true quantized latent for each masked position among distractors. This is a contrastive objective of the kind covered in the article on self-supervised learning. The authors report that after pre-training on 53,000 hours of unlabeled speech, fine-tuning on only ten minutes of labeled audio reaches word error rates of 4.8 and 8.2 percent on the clean and other test sets of LibriSpeech, a public reading-speech benchmark. The lesson for system design is that unlabeled audio is abundant and labeled audio is not, so the representation can be learned from the former and the recognizer fitted with the latter.

Whichever front end is used, the encoder usually reduces the frame rate further. A recognizer does not need 100 vectors per second, because a word takes a few hundred milliseconds. Convolutional subsampling by a factor of 4 or more gives one vector per 40 ms or longer. That reduced frame rate matters twice later: it sets the granularity of the alignment problem in the next section, and it sets the time quantum of the streaming encoder.

Recognition: aligning frames to words

The alignment problem

An encoder turns an utterance into \(T\) frame vectors, and the transcript has \(U\) tokens with \(U\) much smaller than \(T\). Ten seconds of speech at one vector per 40 ms is 250 frames, and the sentence might be 30 characters or 8 word pieces. Nothing in the data says which frames belong to which token, how long each sound lasted, or where the pauses are. Training pairs are (audio, transcript), not (frame, token). A recognizer therefore needs a way to train and decode without labeled alignments. Three families solve this differently, and the choice determines whether the system can stream.

Connectionist temporal classification

Connectionist temporal classification (CTC), introduced by Graves and colleagues in 2006, adds one special symbol, the blank, and lets the model emit one label from the extended vocabulary at every frame. A frame-level sequence of labels is a path \(\pi\) of length \(T\). A collapse function \(\mathcal{B}\) turns a path into an output string by merging adjacent repeats and then deleting blanks. The path aa-a collapses to aa, because the blank separates the two groups of a, while aaa collapses to a single a. The probability of a transcript \(y\) is the sum over every path that collapses to it:

\[ p(y \mid x) = \sum_{\pi \in \mathcal{B}^{-1}(y)} \prod_{t=1}^{T} p(\pi_t \mid x_t) , \]

where \(p(\pi_t \mid x_t)\) is the per-frame softmax output of the network. The training loss is \(-\log p(y \mid x)\). The sum has exponentially many terms, but it can be computed by dynamic programming over the extended label sequence, which has blanks inserted around every label. State \(s\) at time \(t\) can be reached from state \(s\) (stay), from \(s-1\) (advance), and, when the label at \(s\) is not a blank and differs from the label at \(s-2\), from \(s-2\) (skip a blank). The last condition is what forces a blank between two identical labels. The cost is \(O(T \cdot U)\).

Two properties follow from the definition. First, the model needs at least \(U\) frames, and at least \(U + r\) frames when the transcript has \(r\) adjacent repeated labels, because each repeat needs its own blank. Second, the per-frame distributions are conditionally independent given the input. The network cannot condition frame \(t\) on what it emitted at frame \(t-1\), so any dependence between output tokens has to be captured inside the acoustic encoder, or added afterwards by an external language model.

The next program implements the collapse function and the forward recursion for a three-symbol vocabulary and checks the recursion against brute-force enumeration of all paths. The posteriors are random numbers from a seeded generator, not a trained model; the point is the arithmetic.

import itertools
import numpy as np

BLANK = 0
SYMS = {0: "-", 1: "a", 2: "b"}

def collapse(path):
    out, prev = [], None
    for s in path:
        if s != prev and s != BLANK:
            out.append(s)
        prev = s
    return tuple(out)

print("collapse('aa-a') ->", "".join(SYMS[s] for s in collapse((1, 1, 0, 1))))
print("collapse('aaa')  ->", "".join(SYMS[s] for s in collapse((1, 1, 1))))
print("collapse('a-b-') ->", "".join(SYMS[s] for s in collapse((1, 0, 2, 0))))

def ctc_forward(probs, label):
    """P(label | probs) by dynamic programming over the blank-extended label."""
    ext = [BLANK]
    for l in label:
        ext += [l, BLANK]
    T, S = probs.shape[0], len(ext)
    alpha = np.zeros((T, S))
    alpha[0, 0] = probs[0, ext[0]]
    if S > 1:
        alpha[0, 1] = probs[0, ext[1]]
    for t in range(1, T):
        for s in range(S):
            a = alpha[t - 1, s]
            if s >= 1:
                a += alpha[t - 1, s - 1]
            if s >= 2 and ext[s] != BLANK and ext[s] != ext[s - 2]:
                a += alpha[t - 1, s - 2]
            alpha[t, s] = a * probs[t, ext[s]]
    return alpha[-1, -1] + (alpha[-1, -2] if S > 1 else 0.0)

def brute_force(probs, label):
    T, V = probs.shape
    total, n = 0.0, 0
    for path in itertools.product(range(V), repeat=T):
        if collapse(path) == tuple(label):
            n += 1
            total += np.prod([probs[t, s] for t, s in enumerate(path)])
    return total, n

rng = np.random.default_rng(0)
T, V = 6, 3
logits = rng.normal(size=(T, V))
probs = np.exp(logits) / np.exp(logits).sum(1, keepdims=True)
print("frame posteriors (rows = frames; columns = blank, a, b):")
print(np.round(probs, 3))

for label in [(1, 2), (1, 1)]:
    bf, n = brute_force(probs, label)
    dp = ctc_forward(probs, label)
    print("label %-3s: %3d of %d paths collapse to it; brute force %.6f, DP %.6f"
          % ("".join(SYMS[l] for l in label), n, V ** T, bf, dp))

best = tuple(int(i) for i in probs.argmax(1))
print("greedy path:", "".join(SYMS[s] for s in best), "-> output:", "".join(SYMS[s] for s in collapse(best)) or "(empty)")

# length constraint: 'aa' needs at least a blank between the two a's -> 3 frames; 'ab' needs 2
for label in [(1, 1), (1, 2)]:
    for T2 in (1, 2, 3):
        p = np.full((T2, V), 1 / V)
        print("label %s with %d frame(s): P = %.4f" % ("".join(SYMS[l] for l in label), T2, ctc_forward(p, label)))

collapse('aa-a') -> aa
collapse('aaa')  -> a
collapse('a-b-') -> ab
frame posteriors (rows = frames; columns = blank, a, b):
[[0.29  0.224 0.486]
 [0.355 0.187 0.458]
 [0.545 0.382 0.073]
 [0.152 0.288 0.56 ]
 [0.082 0.676 0.242]
 [0.269 0.324 0.407]]
label ab :  70 of 729 paths collapse to it; brute force 0.045678, DP 0.045678
label aa :  35 of 729 paths collapse to it; brute force 0.027610, DP 0.027610
greedy path: bb-bab -> output: bbab
label aa with 1 frame(s): P = 0.0000
label aa with 2 frame(s): P = 0.0000
label aa with 3 frame(s): P = 0.0370
label ab with 1 frame(s): P = 0.0000
label ab with 2 frame(s): P = 0.1111
label ab with 3 frame(s): P = 0.1852

With 6 frames and 3 symbols there are 729 paths. Seventy of them collapse to ab and 35 to aa. The brute-force sum and the dynamic-programming result agree to six decimals for both labels, so the recursion counts every alignment exactly once. The last six lines show the length constraint: the label aa has probability zero with one or two frames and becomes possible only at three (a, blank, a), while ab needs two. The greedy path takes the most probable symbol in each frame and collapses it, giving bbab here. Greedy decoding ignores the sum over alignments, so it is a fast approximation to the best transcript and not its exact maximizer. Beam search over collapsed prefixes, usually with a language model, is the standard improvement.

CTC is simple, trains quickly, and decodes in one left-to-right pass, which makes it attractive for streaming. Its weaknesses come from the independence assumption: it tends to produce outputs that are acoustically plausible but linguistically awkward, such as misspelled words that sound right, unless a language model is fused in.

Transducers

The recurrent neural network transducer (RNN-T), proposed by Graves in 2012, keeps CTC’s alignment-free training and removes its independence assumption. It has three parts: an acoustic encoder producing a vector for each frame, a prediction network that reads the labels emitted so far and acts as an internal language model, and a joint network that combines the two. At each step the joint network gives a distribution over labels plus blank for the pair (frame \(t\), labels emitted \(u\)). Emitting blank advances to the next frame. Emitting a label stays on the same frame and advances \(u\). The model can therefore output several labels for one frame, and the alignment is a path through a \(T \times (U+1)\) lattice. The probability of a transcript is again a sum over paths, computed by the same kind of forward recursion, now over two indices.

The cost is memory during training, since the joint network evaluates a grid of \(T \times U\) positions over the vocabulary, and a more complex decoder. The benefit is that decoding is strictly left to right in time, the label history conditions every decision, and the encoder can be restricted to see only past and a little future audio. For these reasons transducers are a common architecture for streaming recognition. The Conformer encoder of Gulati and colleagues, which interleaves convolution (local patterns) with self-attention (global context), is a frequent choice of acoustic encoder in both CTC and transducer systems.

Attention encoder-decoder models and Whisper

The third family treats recognition as sequence-to-sequence translation from audio to text, exactly as in the article on sequence-to-sequence models and attention. An encoder processes all frames, and an autoregressive decoder produces tokens, attending across the whole encoded input at every step, so that \(p(y \mid x) = \prod_u p(y_u \mid y_{\lt u}, x)\). There is no explicit alignment variable; cross-attention learns it. This model class is strong on accuracy and on tasks that go beyond transcription, but attention is free to look anywhere. A decoder can skip a passage, repeat a phrase, or continue past the end of speech with fluent text that was never said.

Whisper, by Radford and colleagues, is the best-known example. It is an encoder-decoder transformer trained on 680,000 hours of audio paired with transcripts collected from the internet, which the authors call weak supervision because the transcripts are noisy. Its input is an 80-channel log-mel spectrogram computed at 16 kHz with 25 ms windows and a 10 ms stride, over a fixed 30-second segment. The decoder is steered by special tokens that select the language, the task (transcription or translation to English), and whether to predict timestamps. Audio longer than 30 seconds is handled by transcribing consecutive windows and shifting the window forward according to the predicted timestamps. The 30-second window shows the cost of this design for streaming: the encoder is built for complete segments, so using it live means either padding short segments, re-encoding a growing buffer repeatedly, or waiting.

Choosing among them

Property	CTC	RNN-T	Attention encoder-decoder
Alignment	Latent path with blanks, monotonic	Latent path over a frame-label lattice, monotonic	Learned by cross-attention, not constrained to be monotonic
Output dependence	Independent per frame given the input	Conditioned on previous labels via the prediction network	Fully autoregressive over previous tokens
Streaming	Natural with a chunked encoder	Natural with a chunked encoder	Needs the segment, or repeated re-decoding of a growing buffer
Typical failure	Plausible-sounding misspellings without a language model	Higher training memory; more complex decoder	Hallucinated text, repeats, skipped passages
Beyond transcription	Not designed for it	Not designed for it	Natural: translation, language ID, timestamps, prompts

The table is a tendency, not a law: a chunked attention decoder can stream, and a transducer can be trained on very large data. The design question is which failures the application can tolerate. A live captioning system cares about stable partial results and low delay, which favors CTC or a transducer. An offline transcription service for recorded meetings cares about accuracy and punctuation across long audio, which favors the encoder-decoder family.

Decoding and what surrounds the model

The model’s output is rarely the final product. Beam search keeps the several most probable partial transcripts rather than the single best (the sequence-to-sequence article explains beam search and length normalization). A language model trained on text can rescore hypotheses or be fused during the search; this helps CTC most and matters when the audio domain has distinctive vocabulary. Contextual biasing raises the score of expected phrases, such as product names or a caller’s name from a customer record, for the duration of a session. Because it changes scores and not the audio evidence, aggressive biasing causes the system to hear the biased phrase even when it was not spoken.

Recognizers also differ in what their text looks like. Some emit spoken-form text (“order fifteen”), and an inverse text normalization step then writes numbers, dates, and currency in written form (“order 15”). Others emit written form directly. This choice matters for evaluation, covered below, and for downstream parsing. Punctuation, casing, speaker labels (diarization), and word timestamps are separate outputs with separate error rates, and a transcript that reads well can still carry an error in a single number.

Generation: from text to waveform

What text leaves unspecified

Synthesis is a one-to-many problem. The text “order fifteen” does not say how fast to speak, where to place stress, how high the pitch rises, or which voice to use, and the same sentence can be spoken correctly in thousands of ways. Models trained with a regression loss on one recorded rendering tend to average over these possibilities and produce flat, muffled prosody, which is why modern systems add a generative component (an autoregressive decoder, adversarial training, a flow, a diffusion process, or sampling from a token model) to produce one plausible rendering instead of the mean.

The classical pipeline has four stages. Text normalization expands numbers, dates, abbreviations, and symbols into words (“15” becomes “fifteen” for an order number but “one five” for a PIN, which only the context decides). A grapheme-to-phoneme step converts words to phonemes, using a pronunciation dictionary with a learned fallback for unknown words. An acoustic model maps the phoneme or character sequence to a mel spectrogram. A vocoder converts the spectrogram to a waveform. Tacotron 2, by Shen and colleagues, is a representative two-stage system: a recurrent sequence-to-sequence model with attention predicts mel frames from characters, and a modified WaveNet vocoder generates the waveform. The authors report a mean opinion score of 4.53 against 4.58 for professionally recorded speech in their listening test. Scores from different studies, listener pools, and rating protocols are not comparable with each other.

Acoustic models and duration

Autoregressive attention-based acoustic models generate one mel frame at a time and learn the text-to-audio alignment through attention. That alignment can fail: the attention can skip a word, loop on a phrase, or stall, and the failures appear on rare words and long inputs. Non-autoregressive acoustic models avoid this by predicting an explicit duration for each phoneme and expanding the phoneme encodings to the frame rate with a length regulator, then decoding all frames in parallel. The duration predictor is the control surface: scaling durations changes speaking rate, and the alignment cannot skip a word because each phoneme is assigned at least one frame.

The first part of the next program shows the length regulator on a toy example: six phoneme encodings, with random 4-dimensional vectors standing in for learned ones, are repeated by their durations. The second part counts the sequential steps different vocoders need, and the third quantizes synthetic vectors with residual vector quantization, which is introduced below.

Vocoders

A mel spectrogram discards phase, and there are many waveforms with the same magnitude spectrogram. The vocoder must choose one that sounds natural. WaveNet generates each audio sample conditioned on the previous ones, which gives high quality but needs 16,000 to 24,000 sequential neural network evaluations per second of audio. Generative adversarial vocoders such as HiFi-GAN, by Kong and colleagues, produce all the samples of a frame in parallel with transposed convolutions and train against discriminators that look at periodic structure and at spectrogram detail. The authors report 22.05 kHz audio generated about 168 times faster than real time on a V100 GPU, with quality close to human recordings in their listening test. Diffusion and flow-based vocoders typically need several network evaluations per output, which gives back part of the speed advantage; the article on diffusion models explains the sampling cost that this trade-off refers to.

Discrete audio tokens and codec language models

A different design turns audio into a sequence of discrete tokens, so that speech generation becomes the language modeling problem of the earlier articles in this series. A neural audio codec has an encoder, a quantizer, and a decoder. EnCodec, by Défossez and colleagues, is a streaming encoder-decoder with a quantized latent space. Its quantizer is residual vector quantization (RVQ): the first codebook quantizes the encoder output, the second codebook quantizes what the first got wrong, and so on. The reconstruction is the sum of one codebook entry per level:

\[ \hat{z} = \sum_{l=1}^{L} c_l[i_l], \qquad r_l = r_{l-1} – c_l[i_l], \quad r_0 = z , \]

where \(z\) is the encoder output for one frame, \(c_l\) is the codebook at level \(l\), \(i_l\) is the index chosen at that level, and \(r_l\) is the residual still to be explained. Each extra level reduces the error and adds one token per frame. The program below fits RVQ with 64-entry codebooks to 4,000 synthetic correlated 8-dimensional vectors and reports the error after each level. Codebooks are trained with a few k-means iterations on the residuals, which is a simplification of how codecs are trained end to end.

import numpy as np
rng = np.random.default_rng(0)

# 1. Duration-based expansion (length regulator): each phoneme embedding is repeated for its duration in frames.
phones = ["K", "AE", "N", "S", "AH", "L"]            # "cancel", rough ARPAbet-style symbols
emb = rng.normal(size=(len(phones), 4))               # stand-in 4-dim phoneme encodings
durs = np.array([4, 9, 5, 6, 3, 8])                   # frames per phoneme (10 ms hop in this toy)
frames = np.repeat(emb, durs, axis=0)
print("phonemes:", len(phones), "-> mel frames:", len(frames), "(= sum of durations %d)" % durs.sum())
print("speaking faster by 1.25x: durations", np.maximum(1, np.round(durs / 1.25)).astype(int).tolist(),
      "-> frames", int(np.maximum(1, np.round(durs / 1.25)).sum()))

# 2. Sequential steps a vocoder needs per second of audio
sr, hop = 22050, 256
print("\nmel frames per second at %d Hz with hop %d: %.1f" % (sr, hop, sr / hop))
print("autoregressive sample-by-sample vocoder: %d sequential steps per second of audio" % sr)
print("parallel (GAN-style) vocoder: %d samples produced per mel frame, all frames in parallel" % hop)

# 3. Residual vector quantization on synthetic 8-dim 'frame embeddings'
def kmeans(x, k, iters=15):
    c = x[rng.choice(len(x), k, replace=False)]
    for _ in range(iters):
        a = ((x[:, None, :] - c[None]) ** 2).sum(-1).argmin(1)
        for j in range(k):
            if (a == j).any():
                c[j] = x[a == j].mean(0)
    return c

x = rng.normal(size=(4000, 8)) @ rng.normal(size=(8, 8))   # correlated synthetic vectors
K, levels = 64, 6
resid, books, recon = x.copy(), [], np.zeros_like(x)
print("\nresidual vector quantization, codebook size %d, 8-dim synthetic vectors" % K)
print("%7s %20s" % ("levels", "mean squared error"))
print("%7d %20.4f" % (0, (x ** 2).sum(1).mean()))
for L in range(1, levels + 1):
    c = kmeans(resid, K)
    idx = ((resid[:, None, :] - c[None]) ** 2).sum(-1).argmin(1)
    books.append(c)
    recon += c[idx]
    resid = x - recon
    print("%7d %20.4f" % (L, (resid ** 2).sum(1).mean()))

phonemes: 6 -> mel frames: 35 (= sum of durations 35)
speaking faster by 1.25x: durations [3, 7, 4, 5, 2, 6] -> frames 27

mel frames per second at 22050 Hz with hop 256: 86.1
autoregressive sample-by-sample vocoder: 22050 sequential steps per second of audio
parallel (GAN-style) vocoder: 256 samples produced per mel frame, all frames in parallel

residual vector quantization, codebook size 64, 8-dim synthetic vectors
 levels   mean squared error
      0              65.3525
      1              13.7652
      2               4.2969
      3               1.6049
      4               0.6185
      5               0.2465
      6               0.1057

The length regulator turns 6 phonemes into 35 mel frames, which is the sum of the durations, and speaking 1.25 times faster shortens it to 27 frames. At 22.05 kHz with a hop of 256 samples there are about 86 mel frames per second; a sample-by-sample vocoder needs 22,050 sequential steps for each second of audio, while a parallel vocoder produces 256 samples per frame for all frames at once. The RVQ rows show the error falling from 65.35 (no quantization, the signal energy) to 13.77 with one level and 0.11 with six. Each level removes a smaller share of what remains, which is why codecs let the user choose how many codebooks to use and trade bitrate for fidelity. Every level also adds one token per frame, so the token rate grows with the number of codebooks; the multimodal article works through that arithmetic for a hypothetical codec. It is the reason codec language models use a coarse-to-fine structure and not one flat sequence.

VALL-E, by Wang and colleagues, uses this representation. It models the discrete codec codes of speech as a conditional language model, with the phoneme sequence and a three-second recording of an unseen speaker as the prompt, and the authors report that the synthesized speech preserves the speaker’s voice and acoustic environment. The same property is a risk: a system that can imitate a voice from a few seconds of audio needs consent and provenance controls, discussed in the last section. Models that consume and produce audio tokens inside a larger language model, so that the system speaks without an intermediate text step, rely on the cross-modal alignment ideas in the multimodal alignment article.

Where synthesis goes wrong

Synthesis failures differ from recognition failures because the output is a signal that a listener perceives, not a string a program can compare. Normalization errors read a date as a quantity or a code as a number. Pronunciation errors hit names, acronyms, and loanwords that are missing from the lexicon. Alignment failures drop or repeat words. Prosody errors put emphasis on the wrong word, and in a confirmation prompt that can change the meaning: “Cancel order fifteen?” and “Cancel order fifteen.” request different things. Vocoder artifacts appear as buzziness, metallic timbre, or breathing sounds in places with no energy to anchor them. Each of these needs its own test, since a listener-rated overall score averages them away.

Streaming and latency

What latency means for speech

A conversational system has no single latency number. Several delays are measured from different moments, and each one is felt differently by the user.

• Time to first partial result. From the start of speech to the first hypothesis shown or passed on. It governs how responsive live captions feel.
• Finalization latency. From the end of speech to the final transcript. It is the sum of the endpointing decision and the final decoding pass.
• Time to first audio. From the end of the user’s turn to the first audible sample of the reply. It includes recognition, the language model’s time to its first output, synthesis of the first chunk, network transport, and playback buffering. The prefill and decode phases that determine the language model’s contribution are explained in the article on LLM inference optimization.
• Real-time factor (RTF). Compute time divided by audio duration. An RTF below 1 means the stage can keep up with live audio. It says nothing about delay: a stage with an RTF of 0.1 that waits for a 30-second window has a 30-second delay.

A synthesis stage shows why RTF below 1 is necessary for continuous playback. Suppose the first chunk of \(c\) seconds of audio is ready at time \(L_0\) and every later chunk takes \(rc\) seconds to compute, where \(r\) is the RTF. Chunk \(k\) is ready at \(L_0 + (k-1)rc\) and has to start playing at \(L_0 + (k-1)c\). It arrives in time exactly when \(r \le 1\). With \(r = 1\) there is no slack, so any variation in compute or network delay produces an audible gap; practical systems hold a small buffer, which adds to time to first audio. This is the same trade-off as in any real-time media path: buffer size buys robustness to variance at the cost of delay.

Chunked encoders: algorithmic latency

A bidirectional encoder lets every frame attend to every other, which is why it cannot start until the utterance ends. A streaming encoder restricts which frames each position may see. In chunked attention the sequence is cut into chunks of \(C\) frames. A frame attends to all frames in its own chunk, to a limited number of earlier chunks (the left context), and to a few frames of right context, the lookahead. Right context lets a frame use a little of its future, which improves accuracy, and costs delay because the model has to wait for that future to exist.

The delay this design imposes has nothing to do with computing speed. A frame at position \(k\) in a chunk of \(C\) frames, with \(R\) frames of lookahead and frame duration \(\Delta\), cannot be encoded until the chunk and the lookahead have arrived. Measured from the start of the frame, the wait is \((C – k + R)\Delta\) for \(k = 0, \ldots, C-1\). The first frame of a chunk waits longest, \((C + R)\Delta\), and the last waits \((1 + R)\Delta\). The next program prints the attention mask for two settings and tabulates the wait for several chunk and lookahead choices, assuming 40 ms encoder frames. It computes the structural delay only; compute time is separate.

import numpy as np

FRAME_MS = 40          # encoder frame after 4x subsampling of 10 ms frames

def chunk_mask(n, chunk, left_chunks, right_frames):
    """mask[i, j] = True if frame i may attend to frame j."""
    m = np.zeros((n, n), dtype=bool)
    for i in range(n):
        c = i // chunk
        lo = max(0, (c - left_chunks) * chunk)
        hi = min(n, (c + 1) * chunk + right_frames)
        m[i, lo:hi] = True
    return m

m = chunk_mask(12, chunk=4, left_chunks=1, right_frames=0)
print("12 frames, chunk=4, one left chunk, no lookahead (row = query frame, X = visible):")
for row in m:
    print("".join("X" if v else "." for v in row))
m = chunk_mask(12, chunk=4, left_chunks=1, right_frames=2)
print("same, with 2 frames of lookahead:")
for row in m:
    print("".join("X" if v else "." for v in row))

print()
print("%6s %9s %10s %12s %12s" % ("chunk", "lookahead", "chunk ms", "avg wait ms", "worst wait ms"))
for chunk, look in [(1, 0), (4, 0), (8, 0), (4, 2), (8, 2), (16, 4), (None, 0)]:
    if chunk is None:
        # whole-utterance encoder on a 5 s utterance: every frame waits for the end
        n = 5000 // FRAME_MS
        waits = [(n - i) * FRAME_MS for i in range(n)]
        print("%6s %9s %10s %12.0f %12.0f" % ("full", "-", "5000", np.mean(waits), max(waits)))
        continue
    # frame k in a chunk of size c is complete when the chunk (plus lookahead) has arrived
    waits = [(chunk - k + look) * FRAME_MS for k in range(chunk)]
    print("%6d %9d %10d %12.0f %12.0f" % (chunk, look, chunk * FRAME_MS, np.mean(waits), max(waits)))

12 frames, chunk=4, one left chunk, no lookahead (row = query frame, X = visible):
XXXX........
XXXX........
XXXX........
XXXX........
XXXXXXXX....
XXXXXXXX....
XXXXXXXX....
XXXXXXXX....
....XXXXXXXX
....XXXXXXXX
....XXXXXXXX
....XXXXXXXX
same, with 2 frames of lookahead:
XXXXXX......
XXXXXX......
XXXXXX......
XXXXXX......
XXXXXXXXXX..
XXXXXXXXXX..
XXXXXXXXXX..
XXXXXXXXXX..
....XXXXXXXX
....XXXXXXXX
....XXXXXXXX
....XXXXXXXX

 chunk lookahead   chunk ms  avg wait ms worst wait ms
     1         0         40           40           40
     4         0        160          100          160
     8         0        320          180          320
     4         2        160          180          240
     8         2        320          260          400
    16         4        640          500          800
  full         -       5000         2520         5000

The first mask shows the structure: each row is a query frame and each X a frame it may use. Frames 0 to 3 see only their own chunk, frames 4 to 7 see their chunk and the previous one, and no frame sees the future. In the second mask each frame also reads two frames into the next chunk. In the table, a chunk of 4 frames (160 ms) without lookahead has an average wait of 100 ms and a worst case of 160 ms. Adding 2 frames of lookahead moves those to 180 and 240 ms. A chunk of 16 frames with 4 lookahead frames has a worst-case wait of 800 ms. The last row is a whole-utterance encoder on a five-second utterance: the average frame waits 2.5 seconds, which is the reason offline architectures cannot be used live without modification. Larger chunks and more lookahead usually reduce word error rate, because the model has more context for each decision, so the chunk size is a point on an accuracy-latency curve that must be measured on the target audio.

Endpointing

The system also has to decide when the user has finished. Endpointing declares the end of a turn, and a common method detects speech and waits for a fixed amount of silence after the last detected speech, called the hangover. Short hangover means a fast reply and a risk of cutting the speaker off in a pause. Long hangover means a safe turn boundary and a slow system. The cost is asymmetric: cutting someone off sends the model a truncated request, and the user has to repeat it. People pause while they recall a number, and those pauses can be long.

The program below makes this concrete. It builds a synthetic recording of speech-like bursts: 1.2 s of harmonic tone, an inner pause, 0.9 s more tone, then silence, with low-level noise throughout. An energy detector over 20 ms frames marks a frame as speech when its level exceeds -40 dB relative to full scale. The endpoint fires once the hangover has elapsed with no speech after some speech has been heard. The signals are synthetic and the detector is deliberately simple; real systems use trained voice-activity models and often combine them with the recognizer’s own end-of-sentence signal.

import numpy as np

SR, FRAME = 16000, 320          # 20 ms frames
rng = np.random.default_rng(1)

def burst(dur, f0):
    t = np.arange(int(dur * SR)) / SR
    x = sum(np.sin(2 * np.pi * f0 * k * t) / k for k in (1, 2, 3))
    env = 0.5 * (1 - np.cos(2 * np.pi * np.minimum(t / dur, 1.0)))   # smooth on/off
    return 0.3 * x * (0.3 + 0.7 * env)

def silence(dur):
    return 0.003 * rng.normal(size=int(dur * SR))

def make_utterance(inner_pause):
    """words, an inner pause, more words, then trailing silence. Returns audio and true end (s)."""
    parts = [silence(0.4), burst(1.2, 140), silence(inner_pause), burst(0.9, 140), silence(2.0)]
    end = 0.4 + 1.2 + inner_pause + 0.9
    return np.concatenate(parts), end

def endpoint(x, hangover_ms, thresh_db=-40.0):
    n = len(x) // FRAME
    e = np.array([np.mean(x[i * FRAME:(i + 1) * FRAME] ** 2) for i in range(n)])
    db = 10 * np.log10(e + 1e-12)
    speech = db > thresh_db
    need = int(hangover_ms / 20)
    heard, quiet = False, 0
    for i, s in enumerate(speech):
        if s:
            heard, quiet = True, 0
        elif heard:
            quiet += 1
            if quiet >= need:
                return (i + 1) * FRAME / SR      # time the system declares the utterance finished
    return None

for pause in (0.35, 0.70):
    x, end = make_utterance(pause)
    print("inner pause %.2f s, true end of speech at %.2f s" % (pause, end))
    print("  %-10s %-12s %-s" % ("hangover", "endpoint at", "result"))
    for h in (200, 300, 500, 800, 1200):
        ep = endpoint(x, h)
        if ep < end:
            res = "CUT OFF: fired %.2f s before the speaker finished" % (end - ep)
        else:
            res = "ok, response can start %.0f ms after the speech ended" % (1000 * (ep - end))
        print("  %-10s %-12s %s" % ("%d ms" % h, "%.2f s" % ep, res))

inner pause 0.35 s, true end of speech at 2.85 s
  hangover   endpoint at  result
  200 ms     1.80 s       CUT OFF: fired 1.05 s before the speaker finished
  300 ms     1.90 s       CUT OFF: fired 0.95 s before the speaker finished
  500 ms     3.36 s       ok, response can start 510 ms after the speech ended
  800 ms     3.66 s       ok, response can start 810 ms after the speech ended
  1200 ms    4.06 s       ok, response can start 1210 ms after the speech ended
inner pause 0.70 s, true end of speech at 3.20 s
  hangover   endpoint at  result
  200 ms     1.80 s       CUT OFF: fired 1.40 s before the speaker finished
  300 ms     1.90 s       CUT OFF: fired 1.30 s before the speaker finished
  500 ms     2.10 s       CUT OFF: fired 1.10 s before the speaker finished
  800 ms     4.00 s       ok, response can start 800 ms after the speech ended
  1200 ms    4.40 s       ok, response can start 1200 ms after the speech ended

With a 0.35 s inner pause, hangovers of 200 and 300 ms fire in the pause and cut the utterance off, about a second before the speaker actually finished. A 500 ms hangover survives the pause and responds 510 ms after the end of speech. With a 0.70 s pause, 500 ms also fails, and only 800 ms is safe, at a cost of 800 ms of delay in every turn. The endpoint delay is essentially the hangover, plus a frame. No single hangover is right for both speakers, so deployed systems adapt: they shorten the wait when the recognizer’s partial text already looks like a complete sentence or a completed form field, extend it when the text ends mid-phrase or in the middle of a digit sequence, and use semantic end-of-turn models.

The end-to-end budget

A latency budget allocates a target time across every stage on the path. The glossary entry on latency budgets covers the general method, including the point that step percentiles do not add and that overlapping work needs the real dependency path instead of a sum. The next program applies it to the order-desk assistant. The stage costs are assumed design parameters chosen for the example, not measurements of any product, and the program only does the arithmetic of how they combine.

# Assumed stage costs in milliseconds for a hypothetical cascaded voice agent.
# These are design parameters chosen for the example, not measurements of any product.
A = dict(
    capture_chunk=20,       # audio is shipped in 20 ms packets
    uplink=40,              # one-way network, client -> server
    endpoint_hangover=500,  # silence required before the utterance is declared finished
    asr_finalize=80,        # final decoding pass after endpoint
    llm_first_tokens=250,   # time until the first sentence-worth of tokens is available
    tts_first_audio=120,    # time for TTS to emit the first audio chunk of that sentence
    downlink=40,            # one-way network, server -> client
    playout_buffer=60,      # client jitter buffer before playback starts
)

def total(d, keys):
    return sum(d[k] for k in keys)

# Strictly sequential: wait for the whole LLM reply (say 3 sentences, 700 ms more) and the whole TTS (400 ms more).
seq = dict(A)
seq["llm_first_tokens"] = A["llm_first_tokens"] + 700
seq["tts_first_audio"] = A["tts_first_audio"] + 400
keys = list(A)
print("%-18s %10s %12s" % ("stage", "sequential", "streamed"))
for k in keys:
    print("%-18s %10d %12d" % (k, seq[k], A[k]))
print("%-18s %10d %12d" % ("time to first audio", total(seq, keys), total(A, keys)))

print()
print("effect of the endpoint hangover on the streamed total:")
for h in (300, 500, 800):
    d = dict(A, endpoint_hangover=h)
    print("  hangover %4d ms -> %4d ms" % (h, total(d, keys)))

# fraction of the budget the model stages use
share = (A["asr_finalize"] + A["llm_first_tokens"] + A["tts_first_audio"]) / total(A, keys)
print("\nshare of the streamed total spent in the three model stages: %.0f%%" % (100 * share))

stage              sequential     streamed
capture_chunk              20           20
uplink                     40           40
endpoint_hangover         500          500
asr_finalize               80           80
llm_first_tokens          950          250
tts_first_audio           520          120
downlink                   40           40
playout_buffer             60           60
time to first audio       2210         1110

effect of the endpoint hangover on the streamed total:
  hangover  300 ms ->  910 ms
  hangover  500 ms -> 1110 ms
  hangover  800 ms -> 1410 ms

share of the streamed total spent in the three model stages: 41%

In the sequential column the system waits for the language model to finish its entire reply and the synthesizer to finish the entire audio before playing. Time to first audio is 2,210 ms. In the streamed column the language model passes its first sentence to the synthesizer as soon as it is complete and the synthesizer returns the first audio chunk before the rest of the text exists. Time to first audio drops to 1,110 ms with the same stage speeds, because the stages overlap instead of queueing. The endpoint hangover alone is 500 of those milliseconds, 45 percent of the budget, and the three model stages together account for 41 percent. Changing the hangover from 300 to 800 ms moves the total from 910 to 1,410 ms, more than most model-side optimizations would. Measuring each stage separately, with percentiles and not averages, shows which stage to attack.

Partial results, revisions, and barge-in

A streaming recognizer emits partial hypotheses that it later revises. Consumers need to know what is stable. A caption display can tolerate small revisions at the end of the line; a downstream system that triggers an action on a partial transcript cannot. The usual design separates a stable prefix (words the recognizer will not change) from an unstable tail, and an application commits only on stable text or on the final result. The fraction of words in a partial hypothesis that survive unchanged into the final transcript is a useful stability metric to track next to word error rate.

Barge-in is the user speaking while the system is talking. The system has to detect speech in the presence of its own playback, stop synthesis and discard queued audio, and keep the part of the reply that was actually heard in the conversation state, since the user never heard the rest. Acoustic echo cancellation removes the system’s own voice from the microphone signal; without it, the recognizer transcribes the assistant, and the assistant interrupts itself.

Measuring speech systems

Word error rate

Word error rate (WER) aligns the hypothesis to the reference transcript with a minimum edit distance at the word level and counts the edits:

\[ \mathrm{WER} = \frac{S + D + I}{N} , \]

where \(S\) is the number of substituted words, \(D\) the number of deleted words, \(I\) the number of inserted words, and \(N\) the number of words in the reference. The alignment is a dynamic program over a table of prefix distances, the same algorithm as string edit distance with words as symbols. WER is zero for a perfect transcript and has no upper limit of 1, because a hypothesis with many insertions can have more errors than the reference has words. The character error rate (CER) is the same computation over characters and suits languages written without spaces between words.

The next program implements the alignment with a backtrace, so it reports which edits it counted. It then shows three properties that matter in practice: equal WER hides different consequences, normalization decides the number, and the aggregation method changes the result. The transcripts are short synthetic strings, and the bootstrap data at the end are simulated by drawing errors at a fixed per-word rate, so they say nothing about any real recognizer.

import re
import numpy as np

def align(ref, hyp):
    """Word-level Levenshtein with backtrace. Returns (S, D, I, ops)."""
    R, H = len(ref), len(hyp)
    d = np.zeros((R + 1, H + 1), dtype=int)
    d[:, 0] = range(R + 1); d[0, :] = range(H + 1)
    for i in range(1, R + 1):
        for j in range(1, H + 1):
            sub = d[i - 1, j - 1] + (ref[i - 1] != hyp[j - 1])
            d[i, j] = min(sub, d[i - 1, j] + 1, d[i, j - 1] + 1)
    i, j, S, D, I, ops = R, H, 0, 0, 0, []
    while i or j:
        if i and j and d[i, j] == d[i - 1, j - 1] + (ref[i - 1] != hyp[j - 1]):
            if ref[i - 1] != hyp[j - 1]:
                S += 1; ops.append("S(%s>%s)" % (ref[i - 1], hyp[j - 1]))
            i, j = i - 1, j - 1
        elif i and d[i, j] == d[i - 1, j] + 1:
            D += 1; ops.append("D(%s)" % ref[i - 1]); i -= 1
        else:
            I += 1; ops.append("I(%s)" % hyp[j - 1]); j -= 1
    return S, D, I, ops[::-1]

def wer(ref, hyp):
    r, h = ref.split(), hyp.split()
    S, D, I, ops = align(r, h)
    return (S + D + I) / len(r), (S, D, I), ops

pairs = [
    ("do not cancel order fifteen", "do cancel order fifteen"),
    ("do not cancel order fifteen", "do not cancel order fifty"),
    ("do not cancel order fifteen", "do not cancel order fifteen please"),
    ("do not cancel order fifteen", "thank you for watching thank you for watching"),
]
for ref, hyp in pairs:
    w, sdi, ops = wer(ref, hyp)
    print("WER %.2f  (S,D,I)=%s  %s" % (w, sdi, " ".join(ops)))

def normalize(s):
    s = s.lower()
    s = re.sub(r"[^\w\s']", " ", s)
    return " ".join(s.split())

ref = "Order 15, please. I'd like a refund."
hyp = "order fifteen please I would like a refund"
print("\nraw strings        WER = %.2f" % wer(ref, hyp)[0])
print("lowercase+no punct WER = %.2f" % wer(normalize(ref), normalize(hyp))[0])
RULES = {"15": "fifteen", "i'd": "i would"}
def normalize_full(s):
    return " ".join(RULES.get(w, w) for w in normalize(s).split())
print("lowercase+punct+number/contraction rules, applied to both sides: WER = %.2f"
      % wer(normalize_full(ref), normalize_full(hyp))[0])

# corpus-level vs mean of utterance-level WER
utts = [(5, 1), (5, 0), (50, 10), (4, 2)]            # (reference words, errors)
mean_utt = np.mean([e / n for n, e in utts])
pooled = sum(e for _, e in utts) / sum(n for n, _ in utts)
print("\nmean of per-utterance WER %.3f, pooled corpus WER %.3f" % (mean_utt, pooled))

# bootstrap interval on synthetic per-utterance counts for two conditions
rng = np.random.default_rng(7)
def synth(n_utts, rate):
    n = rng.integers(4, 30, n_utts)
    e = rng.binomial(n, rate)
    return n, e
def boot(n, e, B=2000):
    idx = rng.integers(0, len(n), (B, len(n)))
    return np.sort(e[idx].sum(1) / n[idx].sum(1))
print("\nsynthetic test sets (errors drawn at a fixed per-word rate):")
for name, count, rate in [("quiet  ", 400, 0.06), ("noisy  ", 400, 0.09), ("noisy-s", 40, 0.09)]:
    n, e = synth(count, rate)
    b = boot(n, e)
    print("%s %3d utts, %5d words: WER %.3f, 95%% interval [%.3f, %.3f]"
          % (name, count, n.sum(), e.sum() / n.sum(), b[int(0.025 * len(b))], b[int(0.975 * len(b))]))

WER 0.20  (S,D,I)=(0, 1, 0)  D(not)
WER 0.20  (S,D,I)=(1, 0, 0)  S(fifteen>fifty)
WER 0.20  (S,D,I)=(0, 0, 1)  I(please)
WER 1.60  (S,D,I)=(5, 0, 3)  I(thank) I(you) I(for) S(do>watching) S(not>thank) S(cancel>you) S(order>for) S(fifteen>watching)

raw strings        WER = 0.86
lowercase+no punct WER = 0.43
lowercase+punct+number/contraction rules, applied to both sides: WER = 0.00

mean of per-utterance WER 0.225, pooled corpus WER 0.203

synthetic test sets (errors drawn at a fixed per-word rate):
quiet   400 utts,  6795 words: WER 0.059, 95% interval [0.054, 0.065]
noisy   400 utts,  6433 words: WER 0.090, 95% interval [0.083, 0.097]
noisy-s  40 utts,   638 words: WER 0.107, 95% interval [0.084, 0.129]

The first three hypotheses each have a WER of 0.20 against the five-word reference, yet they are different events: dropping “not” reverses the request, changing “fifteen” to “fifty” targets another order, and adding “please” changes nothing that matters. WER counts word edits and does not weigh their consequences. The fourth hypothesis is eight words of fluent text for a five-word reference: three insertions and five substitutions give a WER of 1.60. That is the signature of a hallucinating model, and a mean WER over a test set can be dominated by a few such utterances.

The normalization block shows how much the scoring convention matters. The reference “Order 15, please. I’d like a refund.” and the hypothesis “order fifteen please I would like a refund” contain the same content. Compared as raw strings they score 0.86. Lowercasing and removing punctuation gives 0.43. Applying number and contraction rules to both sides gives 0.00. A reported WER therefore needs the normalizer that produced it, and two systems can only be compared when they were scored with the same one; Whisper’s evaluation applies its own text normalization before computing WER for this reason.

The aggregation block shows two ways to combine utterances. With four utterances of 5, 5, 50, and 4 reference words and 1, 0, 10, and 2 errors, the mean of per-utterance WER is 0.225 while the pooled value, total errors over total words, is 0.203. The mean gives the 4-word utterance the same weight as the 50-word one. Pooling is the usual convention for corpus WER, and the per-utterance distribution is worth reporting separately because it exposes the heavy tail.

The bootstrap block resamples utterances with replacement to put an interval around WER. On the simulated 400-utterance sets, the quiet condition scores 0.059 with an interval of [0.054, 0.065] and the noisy condition 0.090 with [0.083, 0.097]; the intervals do not overlap, so the gap is unlikely to be sampling noise. The 40-utterance noisy set gives a similar point estimate of 0.107 but an interval of [0.084, 0.129], three times wider. Resampling individual utterances assumes they are independent. Utterances from the same speaker or recording session are correlated, so a test set with few speakers should be resampled by speaker, and a slice with few speakers should be reported with that caveat.

Metrics beyond WER

WER is one number for a task that has several. Applications add metrics that target what they need.

• Entity and number accuracy. The fraction of names, digits, dates, and amounts transcribed exactly, on a test set built to contain them. A recognizer with a good overall WER can still fail on digit strings.
• Task accuracy. The share of calls where the downstream action, such as the right order and the right intent, was correct, measured end to end. This is the quantity the business needs, and the error contribution of recognition is only one part of it.
• Diarization error rate. The fraction of time attributed to the wrong speaker, missed speech, or false speech, for systems that label who spoke.
• Timestamp accuracy. The distance between predicted and true word boundaries, which matters for captions and for the redaction procedure in the next section.
• Confidence calibration. Whether a word with a stated confidence of 0.9 is right about 90 percent of the time, which decides when the system should ask the caller to repeat or confirm.

The article on multimodal models discusses evaluation across modalities; for speech the practical consequence is to slice the test set by acoustic condition, speaker group, accent, and vocabulary type, since a single average hides the slices where the system fails.

Evaluating generated speech

There is no reference waveform for a synthesis test: many different waveforms are correct. The usual evaluations are listed here, and each answers a different question.

• Mean opinion score (MOS). Listeners rate samples on a 1 to 5 scale and the mean is reported with a confidence interval. It depends on the listener pool, the rating instructions, the playback conditions, and the other samples in the session, so it is only comparable inside one test. Paired preference tests, where listeners pick the better of two samples, are more sensitive for close systems.
• Intelligibility by round trip. Synthesize a set of sentences, transcribe them with a fixed recognizer, and compute WER or CER against the input text. It is automatic and cheap, and it measures intelligibility as heard by that recognizer, which may be more forgiving than a person on a noisy line.
• Speaker similarity. The cosine similarity between speaker embeddings of the synthesized and reference voices, for voice-matching systems.
• Failure counts. The number of skipped, repeated, truncated, or mispronounced items per 1,000 utterances on a targeted test set of numbers, dates, names, and long inputs. These rare events do not move an average MOS.

Reporting latency

Latency is a distribution, and the user remembers the slow turns. Report percentiles of time to first partial, finalization latency, and time to first audio, over the same set of turns, and keep the failed and timed-out turns in the denominator. Percentiles of individual stages do not add up to the percentile of the whole path, as the latency budget entry shows, so the end-to-end distribution has to be measured directly. Record the conditions as well: network path, device, and concurrent load, because a real-time factor measured on an idle accelerator does not hold under production traffic.

Privacy and failure modes

What audio contains

A recording carries more than the words. Voice is a biometric trait that can identify a person and be matched across recordings. The signal also holds information about health, emotion, age, accent, and location through background sound, and it captures bystanders who never agreed to anything. Transcripts add personal data in plain text: names, addresses, account and card numbers read aloud. The general principle in the glossary entry on data privacy applies directly: collect only what the purpose needs, state the purpose, and control retention. Rules on recording and consent differ across jurisdictions, so a design needs a legal review for each deployment region and not an assumption.

Engineering controls follow from where the audio travels. Processing on the device keeps raw audio local at the cost of model size and update cadence. Server-side processing allows larger models and needs transport security, access control, a decision about whether to retain audio at all, and a clear contract with any provider about retention and training use. Retaining the transcript and discarding the audio removes the biometric content but not the personal data in the words. Deletion has to reach every copy: original audio, transcripts, derived embeddings, logs, evaluation sets, and caches, a problem the article on document pipelines for retrieval treats for text indexes and that applies in the same form to audio stores. Application-level controls on logs, provider retention, and output handling are developed in the article on privacy, data leakage, and output controls.

Redaction by timestamp

A common control is to find personal data in the transcript and remove it from the audio before storage. The word timestamps from the recognizer locate the span. The next program builds a four-second synthetic recording with noise bursts standing in for words and mutes the span of three spoken digits using the reported timestamps. It then assumes the recognizer’s timestamps are 80 ms earlier than the true word positions, and measures how much of each digit’s energy survives the mute.

import numpy as np

SR = 16000
rng = np.random.default_rng(3)
# A 4-second synthetic recording: noise bursts stand in for words (start s, end s).
words = [("my", 0.20, 0.45), ("card", 0.50, 0.90), ("number", 0.95, 1.50),
         ("is", 1.55, 1.70), ("four", 1.80, 2.15), ("two", 2.25, 2.55),
         ("one", 2.65, 2.95), ("thanks", 3.20, 3.70)]
SHIFT = 0.08       # the recognizer's timestamps are assumed to be 80 ms early for every word

def render(shift):
    x = np.zeros(4 * SR)
    for _, s, e in words:
        a, b = int(round((s + shift) * SR)), int(round((e + shift) * SR))
        x[a:b] = 0.2 * rng.normal(size=b - a)
    return x

true_audio = render(SHIFT)               # where the words really are
pii = [w for w in words if w[0] in ("four", "two", "one")]
span = (pii[0][1], pii[-1][2])           # reported timestamps of the digit words

def mute(x, s, e, pad):
    y = x.copy()
    y[int(round(max(0, s - pad) * SR)):int(round((e + pad) * SR))] = 0.0
    return y

print("muted span from reported timestamps: %.2f-%.2f s" % span)
print("%-8s %-8s %s" % ("pad", "word", "share of the word's energy that survives"))
for pad in (0.0, 0.1):
    y = mute(true_audio, *span, pad)
    for w, s, e in pii:
        a, b = int(round((s + SHIFT) * SR)), int(round((e + SHIFT) * SR))
        print("%-8s %-8s %.1f%%" % ("%.2f s" % pad, w, 100 * np.sum(y[a:b] ** 2) / np.sum(true_audio[a:b] ** 2)))

muted span from reported timestamps: 1.80-2.95 s
pad      word     share of the word's energy that survives
0.00 s   four     0.0%
0.00 s   two      0.0%
0.00 s   one      24.9%
0.10 s   four     0.0%
0.10 s   two      0.0%
0.10 s   one      0.0%

With no padding, the first two digits are removed completely, but 24.9 percent of the energy of the last digit survives, because the true word ends 80 ms after the reported end. Padding the muted span by 100 ms on each side removes everything in this example. The general lesson is that redaction inherits the error of the timestamps, so the padding has to be at least as large as the timestamp error measured on the target audio, and the redacted output should be re-transcribed to confirm that the sensitive words are gone. Redacting the transcript alone leaves the audio exposed, and redacting the audio alone leaves the transcript, and anything computed from it, exposed.

Failure modes

The failures below follow from the mechanisms in the earlier sections. The table lists a cause, a way to detect each, and a mitigation.

Failure	Mechanism	Detection	Mitigation
Fabricated text on silence or noise	An autoregressive decoder with a strong language prior continues a transcript when the audio gives no evidence. Koenecke and colleagues found fabricated phrases in about 1 percent of Whisper transcriptions in their study, with higher rates for speakers with aphasia, whose speech contains longer silences.	Compare against a voice-activity detector; flag text with no speech energy; sample audits	Gate decoding on detected speech, trim long silences, use no-speech and repetition thresholds, and review high-stakes output
Dropped negation or wrong number	Short, low-energy or confusable words; no language-model cue because both readings are grammatical	Entity and negation test sets; confidence scores	Read back and confirm before any action that changes state
Poor accuracy for some speakers or conditions	Training data covers accents, dialects, languages, ages, and microphones unevenly	Metrics sliced by group and condition, with intervals	Targeted data collection, adaptation, a fallback channel
Rare words and names recognized as common words	Out-of-vocabulary sounds mapped to likely text	Entity accuracy on domain terms	Contextual biasing from a vetted list, domain adaptation
Premature or late endpoint	Fixed hangover against variable pauses	Cut-off rate and endpoint delay distribution	Adaptive endpointing using partial text and turn context
Unstable partial results	Revision of the unstable tail	Stability metric	Commit only stable text or finals
Skipped or repeated words in synthesis	Attention alignment failure in autoregressive acoustic models	Round-trip WER; per-1,000 failure counts	Duration-based models, length limits, regeneration checks
Wrong spoken form	Normalization ambiguity	Targeted number and date tests	Explicit spoken-form fields, context-aware normalization
Audible gaps	Real-time factor above 1 or insufficient buffer	Underrun counters, percentile of chunk arrival margin	Larger first chunk or buffer, faster vocoder

Security and misuse

Voice has been used as a credential, and synthesis breaks that use. A system that can reproduce a voice from seconds of audio can also produce audio that passes a speaker-verification check, so voice alone is not a strong factor for authorizing consequential actions. Generated speech should carry provenance information where the application allows, such as an embedded watermark or signed metadata, and voice cloning should require a documented consent from the voice owner. None of these controls is complete, because watermarks can be degraded by re-recording and metadata can be stripped, so they reduce risk and do not remove it.

Recognizers are also open to inputs that humans do not perceive as instructions. Carlini and Wagner showed a white-box optimization that takes a given waveform and produces another, over 99.9 percent similar, that DeepSpeech transcribes as any phrase the attacker chooses. A voice interface that executes commands from whatever audio it hears therefore also hears television, other devices, and crafted audio. The practical rules are the same as for any untrusted input: treat the transcript as data and not as authority, require confirmation for actions with side effects, limit what a voice session may do without an additional factor, and keep the audio and decision logs needed to investigate an incident.

References

• Graves, A., Fernández, S., Gomez, F., Schmidhuber, J. (2006). Connectionist Temporal Classification: Labelling Unsegmented Sequence Data with Recurrent Neural Networks. ICML. cs.toronto.edu
• Graves, A. (2012). Sequence Transduction with Recurrent Neural Networks. arXiv:1211.3711
• Gulati, A., et al. (2020). Conformer: Convolution-augmented Transformer for Speech Recognition. arXiv:2005.08100
• Baevski, A., Zhou, H., Mohamed, A., Auli, M. (2020). wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations. arXiv:2006.11477
• Radford, A., et al. (2022). Robust Speech Recognition via Large-Scale Weak Supervision. arXiv:2212.04356
• Shen, J., et al. (2017). Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictions. arXiv:1712.05884
• Kong, J., Kim, J., Bae, J. (2020). HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis. arXiv:2010.05646
• Défossez, A., Copet, J., Synnaeve, G., Adi, Y. (2022). High Fidelity Neural Audio Compression. arXiv:2210.13438
• Wang, C., et al. (2023). Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers. arXiv:2301.02111
• Koenecke, A., et al. (2024). Careless Whisper: Speech-to-Text Hallucination Harms. arXiv:2402.08021
• Carlini, N., Wagner, D. (2018). Audio Adversarial Examples: Targeted Attacks on Speech-to-Text. arXiv:1801.01944