Sampling

In language generation, sampling draws an output token from a probability distribution rather than always selecting the highest-scoring token. This entry concerns output selection, not drawing rows for a statistical study or rebalancing a training dataset.

Change the candidate distribution

With illustrative probabilities 0.6, 0.3, and 0.1, the first token is the most likely but not the only possible draw. Temperature changes how concentrated the distribution is: for positive temperatures, lowering it sharpens relative preferences and raising it flattens them. It does not tell the model which statement is true.

Top-k keeps a fixed number of highest-scoring candidates. Top-p keeps a smallest high-probability set whose cumulative probability reaches the chosen threshold. At top-p 0.8 in this three-token example, the first two reach 0.9 and are retained, then sampling probabilities are renormalized. Hugging Face's generation configuration documents these controls.

Variation and evidence are different issues

Even an unlikely candidate can be correct, and a common continuation can invent a policy. Sampling settings manage selection behavior; correctness needs evidence and evaluation.

A fixed random seed can help reproduce a run in a controlled environment, but it does not guarantee identical results across model versions, hardware, or execution changes. Record the full relevant setup. Greedy selection is a different rule; whether a service accepts zero temperature or maps it to greedy behavior depends on that interface.

References

• Hugging Face: Generation configuration.