Endpointing

Endpointing is the decision that a speaker has finished a turn, so the system can stop listening, finalize the transcript, and respond. In a voice assistant it sits between the microphone and the language model: decide too early and the request is truncated; decide too late and every reply is delayed by the wait.

Why it is hard

Silence is not a reliable signal of completion. People pause to recall a number, to breathe, or while they think about the next clause, and those pauses can be as long as the gap after a finished sentence. The detector sees only a stretch of low energy, and has to guess which kind it is, while the user experiences every extra tenth of a second as lag.

The standard mechanism: voice activity plus hangover

A common design has two pieces.

1. A voice-activity detector (VAD) labels each short frame, typically 10 to 30 ms, as speech or non-speech. The simplest VAD compares frame energy with a threshold, for example a level above -40 dB relative to full scale; trained VAD models are more robust to noise.
2. The endpointer fires when speech has been heard and non-speech has then lasted for the hangover duration, the amount of trailing silence required before the turn is declared over.

Trace on 100 ms frames, where 1 means speech: 0 0 1 1 1 1 0 0 0 1 1 1 0 0 0 0 0 0 0 0. The speaker says something, pauses for three frames (300 ms), and continues. With a 200 ms hangover the endpoint fires at the end of frame 7, in the middle of the pause (t = 0.8 s), and the second half of the sentence is lost. With 300 ms it fires at the end of frame 8, still inside the pause (t = 0.9 s), because the pause is exactly three frames long. With 500 ms it fires after the real end of speech, at the end of frame 16 (t = 1.7 s), and the pause is correctly ignored. The price is that the system now always waits half a second after the final word.

The trade-off

The hangover \(h\) adds exactly \(h\) of latency to every turn, and it removes cut-offs only for pauses shorter than \(h\). If the inner pauses of an utterance are distributed with a long tail, the cut-off rate falls slowly as \(h\) grows. A hypothetical simulation draws 3 inner pauses per utterance from a lognormal distribution with a median of 0.35 s and log-space standard deviation 0.6. The fixed-seed simulation generates 100,000 utterances and counts a cutoff when any pause is at least \(h\), matching the frame trace:

import numpy as np
rng = np.random.default_rng(0)
pauses = rng.lognormal(mean=np.log(0.35), sigma=0.6, size=(100000,3))
for h in (0.3,0.5,0.7,1.0,1.5):
    cutoff = (pauses >= h).any(axis=1).mean()
    print(f"{h:.1f} s  {100*cutoff:.1f}%  {h:.1f} s")

hangover	utterances cut off	added latency
0.3 s	93.8%	0.3 s
0.5 s	62.3%	0.5 s
0.7 s	32.8%	0.7 s
1.0 s	11.5%	1.0 s
1.5 s	2.3%	1.5 s

The numbers depend entirely on the assumed pause distribution, but the shape is general: halving the cut-off rate costs a large increase in latency, and the two errors are not equally priced. A cut-off sends a truncated request downstream and forces the user to repeat it, which usually costs more than a fraction of a second.

Ways to do better than a fixed hangover

• Use the recognizer’s own state. If the partial transcript ends mid-phrase (“my account number is”), wait longer; if it forms a complete sentence, wait less. Many streaming recognizers also emit an end-of-sentence or end-of-turn signal.
• Use prosody and context, such as falling pitch, or the type of prompt just asked (a spoken digit string tolerates long pauses).
• Separate the endpoint from the commit. The system can start generating at a short hangover and cancel the result if speech resumes, trading compute for latency.
• Allow barge-in: when the user speaks during the reply, stop playback and re-listen, which makes a premature endpoint cheaper to recover from.

Neighbors

VAD answers “is there speech in this frame”. Endpointing answers “is the turn over”, which needs VAD plus a rule about time and usually context. Turn-taking and barge-in cover the interaction after the endpoint.