Token

A token is a unit in the representation a language model processes. For text, it may correspond to a word, a word fragment, punctuation, or a byte-based piece. Special tokens can also mark boundaries or control information. A token identifier is an integer that refers to an entry in a particular tokenizer's vocabulary.

A word is not a fixed number of tokens

Imagine one tokenizer storing a common word as one unit while another splits it into several pieces. Both can encode the same visible text. Spaces, spelling, language, and the tokenizer's learned vocabulary affect the result. There is no universal conversion from words or characters to tokens.

Hugging Face describes word, character, and subword approaches to tokenization. Exact boundaries should be measured with the tokenizer used by the model, not guessed from an English word-count rule.

Identifiers and meaning are different

A token ID of 42 does not mean the number forty-two, nor does it represent the same unit in every vocabulary. The model uses token representations in context; an ID itself is not a semantic explanation.

Token counts matter when allocating context and output length. They do not measure how much reliable knowledge the model has acquired. A one-token answer can be wrong, and a long answer can contain unsupported detail. Tokenization produces the units; generation selects output units.

Reference: Hugging Face: Tokenization algorithms.