With the adoption of retrieval-augmented generation (RAG), large language models (LLMs) are expected to ground their generation to the retrieved contexts. Yet, this is hindered by position bias of LLMs, failing to evenly attend to all contexts. Previous work has addressed this by synthesizing contexts with perturbed positions of gold segment, creating a position-diversified train set. We extend this intuition to propose consistency regularization with augmentation and distillation. First, we augment each training instance with its position perturbation to encourage consistent predictions, regardless of ordering. We also distill behaviors of this pair, although it can be counterproductive in certain RAG scenarios where the given order from the retriever is crucial for generation quality. We thus propose CORD, balancing COnsistency and Rank Distillation. CORD adaptively samples noise-controlled perturbations from an interpolation space, ensuring both consistency and respect for the rank prior. Empirical results show this balance enables CORD to outperform consistently in diverse RAG benchmarks.

翻译：随着检索增强生成（RAG）的采用，大型语言模型（LLMs）被期望基于检索到的上下文进行生成。然而，LLMs的位置偏差阻碍了这一点，使其无法均匀关注所有上下文。先前的研究通过合成黄金片段位置扰动的上下文，创建位置多样化的训练集来解决此问题。我们扩展这一思路，提出通过增强与蒸馏实现一致性正则化。首先，我们对每个训练实例进行位置扰动增强，以鼓励无论顺序如何都能保持一致的预测。我们还对这一对实例的行为进行蒸馏，尽管在某些RAG场景中，检索器提供的顺序对生成质量至关重要，这可能会适得其反。因此，我们提出CORD，以平衡一致性与排序蒸馏。CORD从插值空间中自适应地采样噪声控制的扰动，确保同时满足一致性和对排序先验的尊重。实证结果表明，这种平衡使CORD在多种RAG基准测试中均能持续超越现有方法。