Recently, retrieval augmentation and tool augmentation have demonstrated a remarkable capability to expand the internal memory boundaries of language models (LMs) by providing external context. However, internal memory and external context inevitably clash, leading to knowledge conflicts within LMs. In this paper, we aim to interpret the mechanism of knowledge conflicts through the lens of information flow, and then mitigate conflicts by precise interventions at the pivotal point. We find there are some attention heads with opposite effects in the later layers, where memory heads can recall knowledge from internal memory, and context heads can retrieve knowledge from external context. Moreover, we reveal that the pivotal point at which knowledge conflicts emerge in LMs is the integration of inconsistent information flows by memory heads and context heads. Inspired by the insights, we propose a novel method called Pruning Head via PatH PatcHing (PH3), which can efficiently mitigate knowledge conflicts by pruning conflicting attention heads without updating model parameters. PH3 can flexibly control eight LMs to use internal memory ($\uparrow$ 44.0%) or external context ($\uparrow$ 38.5%). Moreover, PH3 can also improve the performance of LMs on open-domain QA tasks. We also conduct extensive experiments to demonstrate the cross-model, cross-relation, and cross-format generalization of our method.

翻译：近期，检索增强与工具增强技术通过提供外部上下文，显著扩展了语言模型（LM）的内部记忆边界。然而，内部记忆与外部上下文不可避免地产生冲突，导致LM内部出现知识冲突。本文旨在通过信息流的视角解读知识冲突的机制，并通过对关键节点进行精准干预来缓解冲突。我们发现，在模型的后期层中存在若干具有相反效应的注意力头：记忆头可从内部记忆中召回知识，而上下文头则能从外部上下文中检索知识。此外，我们揭示出，LM中知识冲突产生的关键节点在于记忆头与上下文头对不一致信息流的整合。受此启发，我们提出一种名为PH3（Pruning Head via PatH PatcHing）的新方法，该方法通过修剪冲突注意力头来高效缓解知识冲突，而无需更新模型参数。PH3可灵活控制八个LM，使其分别更倾向于使用内部记忆（提升44.0%）或外部上下文（提升38.5%）。此外，PH3还能提升LM在开放域问答任务中的表现。我们进行了大量实验，证明了该方法在跨模型、跨关系及跨格式场景中的泛化能力。