Natural Language Processing (NLP)

Natural language processing is the work of getting computers to handle human language — classifying it, extracting from it, translating it, answering questions with it, generating it. Its central challenge is relating linguistic form and context to the intended task, including ambiguity, composition, and meaning that depends on the situation.

For many text models, tokenization divides the input into units that can be encoded numerically. It determines which distinctions enter the model, but representation capacity, training data, and the learning objective also shape what the system can distinguish.

Representing text without losing the needed information

A word-level vocabulary assigns identifiers to recognized words and commonly uses an unknown marker for missing entries. Character or byte representations use smaller basic units but create longer sequences. Subword tokenizers balance these costs by retaining frequent pieces and splitting other strings. For an illustrative vocabulary containing “un”, “help”, and “ful”, “unhelpful” can be represented by three pieces even if the complete word was not a vocabulary entry. That spelling capability does not imply that the model understands the word.

The exact segmentation depends on the algorithm and vocabulary. BPE learns merge rules, while Unigram uses a probabilistic inventory of pieces; a hand-written longest-match splitter is not interchangeable with either. A complete byte fallback can represent arbitrary encoded text, whereas a vocabulary of only observed characters can still encounter unknown characters. Whitespace, punctuation, normalization, and special tokens also affect whether the original distinctions survive. The Hugging Face tokenizer overview describes these alternatives.

Token identifiers are indices, not semantic magnitudes: token 100 is not twice the meaning of token 50. The model maps them to features or vectors. A tokenizer used with a pretrained model must retain the expected identifier mapping and conventions; changing the vocabulary without adapting the corresponding model components changes the input interpretation.

Token counts differ from word counts and depend on language and domain. A language poorly covered by a tokenizer may require more pieces, using more of a fixed token context and more computation, but measure that effect for the actual tokenizer and text. Truncation is also a modeling decision: if an important exception appears at the end of a document, shortening it can change the answer even when the remaining words are represented perfectly.

The task decides how representations are used

A support-ticket classifier maps a whole message to a category. An entity extractor returns spans such as names or order numbers, so it must preserve offsets into the text and distinguish the same word in different roles. Retrieval ranks documents for a query. Translation or summarization produces a sequence rather than a single label, requiring a decoding procedure and a definition of acceptable output. These tasks can share a language representation while needing different labels, heads, training objectives, and evaluations.

A practical pipeline first defines that output contract, obtains examples, splits evaluation data to avoid duplicates and shared-document leakage, and chooses a representation and model. A narrow classification task may work well with word or n-gram features and a simple classifier. A contextual encoder can help when word sense and relationships matter. A generative model can produce flexible text, but flexibility adds the problem of checking whether the output obeys constraints and is supported by the available information.

From counts to context

Once text is tokens, the history of NLP is the history of what the tokens get turned into. Statistical methods counted them: n-grams, TF-IDF, and hidden Markov models for sequence labeling. Word embeddings replaced counts with learned vectors, giving each type one fixed vector regardless of use.

The decisive change was making the vector depend on the sentence. One fixed type vector assigns the same representation to bank in “river bank” and “bank account.” A contextual representation can differ between these uses because neighboring words affect it. A downstream model can also use context with static input embeddings; the fixed vector alone does not resolve the sense. Recurrent models did this sequentially, and transformers allow parallel processing of known positions during training, subject to attention masks. Autoregressive generation still produces successive output tokens in sequence in its basic form.

The pattern that came out of it — pretrain on unlabeled text, then adapt to a task — makes reuse of pretrained representations possible. Whether to freeze, fine-tune, or use a model through prompting depends on the task and evidence. Reusing a representation presents related choices in vision.

What stays hard

Ambiguity is the standing difficulty and it is structural, not a gap in the data. Words have multiple senses, sentences have multiple parses, pronouns need a referent, and negation and scope invert meaning with one token. A system that gets the average case right can be confidently wrong on a sentence whose meaning turns on a single word — which is why NLP evaluation needs targeted cases and not only aggregate scores.

Evaluate according to the output. Classification needs error analysis by class and relevant language or domain. Span extraction needs correct boundaries as well as types. Retrieval needs relevant items near the positions users will inspect. Generated text needs checks for meaning, factual support, omissions, and required format; overlap with one reference wording cannot establish all of these. Targeted cases for negation, ambiguity, long-range references, and language variation expose weaknesses hidden by a single aggregate score.

Training text can carry stereotypes and uneven language coverage, while annotation rules, objectives, and deployment decisions can introduce further disparities. Measure the effects relevant to the application rather than assuming that one dataset statistic explains every failure. More model capacity can improve some errors but is not a guarantee of fairness, factual accuracy, or faithful explanations. A fluent answer can still be unsupported, and a model-generated rationale is not automatically a record of its actual computation.

When performance changes, inspect the complete path: input decoding and normalization, tokenization and truncation, representation, task head or decoding rule, and the evidence used to score the output. A tokenizer mismatch needs a different remedy from missing domain knowledge or a poorly specified label. This separation makes the error actionable without attributing every language failure to model size.

References

• Sennrich, R., Haddow, B., and Birch, A. (2016). Neural Machine Translation of Rare Words with Subword Units. arXiv:1508.07909. arxiv.org
• Hugging Face documentation, Tokenizers summary. huggingface.co
• Python documentation, collections.Counter. docs.python.org