Tokenization

Tokenization converts text into a sequence of units that can be mapped to token identifiers. A tokenizer may normalize text, find boundaries, divide unfamiliar words into smaller pieces, and add special tokens required by the model. The exact procedure belongs to the chosen tokenizer.

Follow the representation change

Take a short sentence with a product name and a number. A whitespace split would separate visible words, but a subword tokenizer can divide the product name further and represent spaces or digits in its own way. This example describes the process; it does not claim a particular token sequence without running that tokenizer.

Algorithms such as byte-pair encoding, WordPiece, and Unigram construct or use subword vocabularies differently. Hugging Face's tokenizer overview explains these families. Their shared purpose is to produce a usable representation, not to determine whether the sentence is true.

Use the tokenizer that matches the model

If the same ID refers to different pieces in two vocabularies, feeding one tokenizer's IDs to an incompatible model changes the input's meaning. Keep the tokenizer version, special-token rules, and chat formatting aligned with the model.

Converting IDs back to text is often also called decoding. That operation differs from generation decoding, which chooses output tokens from model scores. Normalization and cleanup may change the recovered text, so do not assume every tokenizer is an exact byte-preserving round trip.

Reference: Hugging Face: Tokenization algorithms.