Knowledge Conflict

A knowledge conflict occurs when two sources of information that a language model application uses disagree. Xu et al. distinguish three kinds in a survey:

• Context-memory conflict: the supplied context disagrees with what the model learned in training. The model may answer from memory and ignore the evidence.
• Inter-context conflict: passages in the context disagree with each other.
• Intra-memory conflict: the model’s own knowledge is inconsistent, so it answers the same question differently depending on phrasing.

For retrieval-augmented applications the first two matter most, because the application controls the context and can act before generation.

Check scope, version, and authority first

Passages that appear to disagree can differ in something the answer must account for:

• Version: an older document was superseded by a newer one.
• Scope: both are correct for different groups, places or products (employees get 180, contractors 120).
• Authority: a policy and an informal wiki page disagree, and the policy governs.
• Time of validity: each was true during a different period.

A genuine contradiction remains when equally authoritative sources give different values for the same fact with the same scope and date. Treating every disagreement as a contradiction wastes the information that resolves it, and picking one passage at random hides the others.

Resolution policies

Resolution is best done by rules over metadata before the model sees the passages, because a model has no reliable way to know that a document is superseded unless the text says so, and between two fluent passages it may favor one by position or wording. Typical rule order: filter by scope (does it apply to this requester?), then by validity date, then rank by source authority, then by recency among equals. The order is a policy decision, and different orders pick different answers. The hypothetical program below compares two orderings over four passages that state a limit (id, value, year, authority rank where higher is more authoritative): c1 (120, 2022, 2), c2 (180, 2024, 2), c3 (150, 2024, 1), c4 (100, 2020, 3).

Ordering	Chosen
newest first, authority breaks ties	c2, 180
authority first, recency breaks ties	c4, 100
two passages of equal authority and date, 180 and 150	tie: no rule decides

The same data gives 180 or 100 depending on whether newer or more authoritative wins, which is why the ordering is written down and tested instead of left implicit. The case of a true tie must have its own outcome.

Handling what rules leave open

• Surface the disagreement. Show each source with attribution and date and let the user decide, or say that sources disagree and name them.
• Escalate to the owner of the documents when the answer drives a consequential action.
• Abstain rather than average or guess. A single confident value is wrong in at least one of the two cases.
• Instruct the model to report disagreement among the supplied passages and cite both, then verify the output. This is weaker than metadata rules and is tested on deliberately conflicting passages.

For context-memory conflicts, instruct the model to answer only from the supplied evidence and verify that answer statements are supported by the cited passages. A model that restates a memorized figure from before a policy change is detected by a support check, not by a better prompt alone.

Testing

Build a small set of questions with planted conflicts of each kind (superseded, scope, authority, genuine tie), and check three things: the answer follows the intended policy, a genuine tie is reported as a disagreement, and a conflicting passage that is dropped is recorded in the trace with the reason.

Related terms

Knowledge conflict is about disagreement between sources. It differs from missing evidence, which calls for abstention, and from hallucination, where nothing in the evidence or memory supports the claim. Verification with Natural Language Inference can detect contradiction between two passages, though it cannot decide which one is right.

References

• Xu et al., Knowledge Conflicts for LLMs: A Survey — https://arxiv.org/abs/2403.08319