Natural Language Inference

Natural language inference (NLI), also called recognizing textual entailment, is the task of deciding how a hypothesis relates to a premise. Given two pieces of text, a classifier outputs one of three labels:

• Entailment: if the premise is true, the hypothesis must be true.
• Contradiction: if the premise is true, the hypothesis must be false.
• Neutral: the premise neither guarantees nor rules out the hypothesis.

Premise: "The deadline to request a refund is 30 days from delivery." Hypothesis "Refunds can be requested within 30 days" is entailed. “The deadline to request a refund is 60 days from delivery” contradicts it. "Refunds are processed by the finance team" is neutral, because the premise says nothing about who processes them. The judgment is about what the premise supports, not about what is true in the world.

How models are built

NLI datasets such as SNLI (Bowman et al.) and MultiNLI (Williams et al.) contain hundreds of thousands of premise-hypothesis pairs with human labels. A model is typically a transformer that reads the pair jointly, in the same way as a cross-encoder, and outputs probabilities over the three labels. The same model can be applied outside the original data: any text can be a premise and any claim a hypothesis, which is how NLI became a general-purpose tool for checking text against text.

Use as a support checker

A grounded answer is one whose statements are supported by supplied passages. Support is an entailment relation, so an NLI model can serve as an automatic judge: premise is the cited passage (or the concatenation of cited passages), hypothesis is the statement from the answer. A decision rule is then applied, for example "supported if P(entailment) exceeds a threshold". The ALCE benchmark by Gao et al. used an NLI model for this purpose, and the result feeds metrics such as citation precision and recall. Other uses are detecting contradictions between two passages and checking that a summary does not assert something its source does not.

Limits

An NLI model is a learned classifier and can be wrong in specific, predictable ways.

• Lexical shortcuts. Models can lean on word overlap, and negation, numbers and qualifiers are hard. A trivial overlap score shows the trap. With premise “The deadline to request a refund is 30 days.”, the hypothesis “The deadline to request a refund is not 30 days.” shares 90% of its distinct words with the premise, more than the entailed paraphrase “Refunds can be requested within 30 days.” (29%). These percentages count the fraction of distinct hypothesis words appearing in the premise, after lowercasing and removing punctuation. A request deadline says when an application must be made, not when the money must be paid. A judge that rewarded overlap would prefer the contradiction. Trained models do much better than overlap, and similar errors remain on negation, quantifiers and numbers.
• Premise length. Models are trained mostly on sentence-length premises. Long passages, tables and multi-hop reasoning across several passages are harder, and the passage may be truncated.
• Domain shift. A model trained on image captions and general text may mis-handle legal, medical or policy language.
• Calibration. The probabilities are not guaranteed to match the frequency of correct judgments. A threshold should be picked from labeled examples from the application.
• Entailment is not truth. The check says the passage supports the claim, and cannot tell whether the passage is current or correct.

Practical use

Use NLI as one layer in verification: cheap exact checks (citation IDs exist, quoted spans appear in the cited passage, numbers match) first, then NLI for the support judgment, and a sample of human review to measure the judge's own error rate. A large language model prompted to judge support is an alternative with different costs and failure modes, and it also needs validation against human labels.

Related terms

NLI classifies a pair of texts and is not a fact checker, which would also consult external knowledge. It differs from semantic similarity, where "The deadline is 30 days" and "The deadline is 60 days" are close even though they contradict. A cross-encoder is the usual architecture for NLI classifiers.

References

• Bowman et al., A large annotated corpus for learning natural language inference (SNLI)
• Williams, Nangia and Bowman, A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference (MultiNLI)