Citation Precision and Recall

An answer that cites sources makes two kinds of claims: that each statement is supported, and that the cited passage is where the support lies. Citation precision and citation recall are two metrics that measure these claims separately. They are different from retrieval precision and recall, which measure whether the retriever found relevant documents. Citation metrics look at the finished answer: its statements and the passages attached to them.

Definitions

Split an answer into statements (often sentences). Each statement has a set of cited passages, possibly empty. Support is judged by entailment: the cited passages, read together, say or imply the statement. Gao et al. use this setup in ALCE, a benchmark and set of automatic metrics for answers with citations, judging entailment with a natural language inference model.

• Citation recall is the share of statements that are fully supported by the passages cited for them. An uncited statement, or one whose citations do not entail it, counts against recall. Low recall means unsupported or uncited claims.
• Citation precision is the share of citations that are not irrelevant. In the ALCE formulation, the cited passages must first jointly entail the whole statement; otherwise its citations receive no precision credit. Once that condition holds, a citation is irrelevant when it does not support the statement alone and removing it would not break the support the remaining citations provide. Low precision means decorative or misdirected citations: they look like evidence and are not.

\[ \text{recall}=\frac{\#\{\text{statements entailed by their cited passages}\}}{\#\{\text{statements}\}},\qquad \text{precision}=\frac{\#\{\text{citations that are not irrelevant}\}}{\#\{\text{citations}\}} \]

Both can be computed per answer and then averaged.

A worked example

Hypothetical answer with four statements. Judgments of entailment are given by hand for this example.

Statement	Cited passages	Cited passages together entail it?	Judgment of each citation
s1	p1	yes	p1 supports alone: relevant
s2	p2, p3	yes	p2 supports alone: relevant. p3 does not, and p2 is enough without it: irrelevant
s3	p4	no	p4 does not support, and nothing else supports: irrelevant
s4	none	no	no citation to judge

Citation recall is 2 of 4 statements, 0.5: s1 and s2. Citation precision is 2 of 4 citations, 0.5: p1 and p2 count and p3 and p4 do not. The metrics reach the same value here and can differ: an answer that cites ten passages on every statement has high recall and low precision, and an answer with a single correct citation for half its statements has high precision and low recall.

What each metric protects against

Recall guards against claims the model made without evidence, the property a grounded answer is meant to have. Precision guards against decoration: a citation marker that sends the reader to a passage that does not contain the claim wastes their time and falsely signals that the claim was checked. Neither measures whether the cited passage itself is correct, current or the best available, and neither measures whether the answer is correct. A statement can be perfectly supported by a superseded document.

Conditions and limits

• The metrics inherit the errors of the entailment judge. An NLI model can accept a passage that shares words with the statement and still contradicts it, so the metric is validated against human judgments on a sample from the application’s domain.
• The statement boundary matters. A long sentence with three claims and one citation can pass or fail depending on whether it is judged as one unit.
• A statement composed from two passages needs the passages judged together, which is why support is evaluated on the concatenation of the cited passages and not per passage.
• Studies of deployed systems show the gap is not small: Liu, Zhang and Liang audited four generative search engines in 2023 and found that about half of the generated sentences were fully supported by their citations.

Related terms

Citation precision and recall measure answer-to-source support. They differ from context precision and recall, which measure the retriever’s output, and from answer correctness, which compares the answer with a reference. See also Natural Language Inference, the usual judge of support.

References

• Gao, Yen, Yu and Chen, Enabling Large Language Models to Generate Text with Citations — https://arxiv.org/abs/2305.14627
• Rashkin et al., Measuring Attribution in Natural Language Generation Models — https://arxiv.org/abs/2112.12870