Decoding

In text generation, decoding is the procedure that turns model scores into an output sequence. An autoregressive model scores possible continuations; a decoding strategy selects tokens and determines how generation proceeds or stops. This meaning differs from a tokenizer's operation that converts token IDs back into readable text.

Scores do not choose themselves

Suppose three possible next tokens have probabilities 0.6, 0.3, and 0.1. Greedy decoding chooses the first. Sampling can choose any of them according to the configured distribution. Beam search maintains several candidate sequences instead of only one local choice. These are illustrative probabilities, not outputs from a model test.

A locally most likely token need not lead to the most likely complete sequence. Likewise, a high-scoring sequence is not necessarily factually correct or useful. Hugging Face documents several generation strategies and their controls.

Define constraints and completion

The procedure may stop at an end marker, a configured stop condition, or a length limit. Hitting a limit can leave a sentence or structured object incomplete. Constraints can restrict outputs to a format, but valid syntax does not establish that the values are correct.

Record the model, prompt, decoding settings, and relevant runtime details when comparing results. Changing decoding alters output selection without necessarily changing model weights. A more deterministic strategy can reduce variation while consistently producing the same unsupported answer.

Reference: Hugging Face: Generation strategies.