Large language models (LLMs) are susceptible to generating hallucinated information, despite the integration of retrieval-augmented generation (RAG). Parallel context extension (PCE) is a line of research attempting to effectively integrating parallel (unordered) contexts, while it still suffers from hallucinations when adapted to RAG scenarios. In this paper, we propose DePaC (Dehallucinating Parallel Context Extension), which alleviates the hallucination problem with context-aware negative training and information-calibrated aggregation. DePaC is designed to alleviate two types of in-context hallucination: fact fabrication (i.e., LLMs present claims that are not supported by the contexts) and fact omission (i.e., LLMs fail to present claims that can be supported by the contexts). Specifically, (1) for fact fabrication, we apply the context-aware negative training that fine-tunes the LLMs with negative supervisions, thus explicitly guiding the LLMs to refuse to answer when contexts are not related to questions; (2) for fact omission, we propose the information-calibrated aggregation which prioritizes context windows with higher information increment from their contexts. The experimental results on nine RAG tasks demonstrate that DePaC significantly alleviates the two types of hallucination and consistently achieves better performances on these tasks.

翻译：大型语言模型（LLMs）容易生成包含幻觉的信息，尽管已集成检索增强生成（RAG）。并行上下文扩展（PCE）是一系列旨在有效整合并行（无序）上下文的研究方向，但在适应RAG场景时仍受幻觉问题困扰。本文提出DePaC（消除幻觉的并行上下文扩展），其通过上下文感知的负训练与信息校准的聚合来缓解幻觉问题。DePaC旨在缓解两种类型的上下文内幻觉：事实捏造（即LLMs提出上下文不支持的主张）与事实遗漏（即LLMs未能提出上下文可支持的主张）。具体而言，（1）针对事实捏造，我们采用上下文感知的负训练，通过负监督对LLMs进行微调，从而明确引导LLMs在上下文与问题无关时拒绝回答；（2）针对事实遗漏，我们提出信息校准的聚合方法，该方法优先选择从其上下文中获得更高信息增量的上下文窗口。在九个RAG任务上的实验结果表明，DePaC显著缓解了这两种类型的幻觉，并在这些任务上持续取得了更好的性能。