Retrieval-Augmented Generation (RAG) has become a widely adopted approach to enhance Large Language Models (LLMs) by incorporating external knowledge and reducing hallucinations. However, noisy or irrelevant documents are often introduced during RAG, potentially degrading performance and even causing hallucinated outputs. While various methods have been proposed to filter out such noise, we argue that identifying irrelevant information from retrieved content is inherently difficult and limited number of transformer layers can hardly solve this. Consequently, retrievers fail to filter out irrelevant documents entirely. Therefore, LLMs must be robust against such noise, but we demonstrate that standard fine-tuning approaches are often ineffective in enabling the model to selectively utilize relevant information while ignoring irrelevant content due to the structural constraints of attention patterns. To address this, we propose a novel fine-tuning method designed to enhance the model's ability to distinguish between relevant and irrelevant information within retrieved documents. Extensive experiments across multiple benchmarks show that our approach significantly improves the robustness and performance of LLMs.

翻译：检索增强生成（RAG）已成为一种广泛采用的方法，通过整合外部知识并减少幻觉来增强大型语言模型（LLMs）。然而，在RAG过程中常常会引入噪声或无关文档，这可能导致性能下降，甚至产生幻觉输出。尽管已有多种方法被提出以过滤此类噪声，我们认为从检索内容中识别无关信息本身具有固有难度，且有限数量的Transformer层难以解决此问题。因此，检索器无法完全过滤掉无关文档。鉴于此，LLMs必须具备对此类噪声的鲁棒性，但我们证明，由于注意力模式的结构性限制，标准的微调方法通常无法有效使模型选择性地利用相关信息而忽略无关内容。为解决这一问题，我们提出了一种新颖的微调方法，旨在增强模型在检索文档中区分相关信息与无关信息的能力。在多个基准测试上进行的大量实验表明，我们的方法显著提升了LLMs的鲁棒性和性能。