Long-context models (LCMs) have demonstrated great potential in processing long sequences, facilitating many real-world applications. The success of LCMs can be attributed to their ability to locate implicit critical information within the context for further prediction. However, recent research reveals that LCMs are often susceptible to contextual noise, i.e., irrelevant tokens, that can mislead model attention. In this paper, we conduct a fine-grained analysis of the context noise and propose an effective metric, the Integrated Gradient (IG) score, to detect and quantify the noise information within the context. Our findings reveal that even simple mitigation of detected context noise can substantially boost the model's attention on critical tokens and benefit subsequent predictions. Building on this insight, we propose Context Denoising Training (CDT), a straightforward yet effective training strategy that improves attention on critical tokens while reinforcing their influence on model predictions. Extensive experiments across four tasks, under both context window scaling and long-context alignment settings, demonstrate the superiority of CDT. Notably, when trained with CDT, an open-source 8B model can achieve performance (50.92) comparable to GPT-4o (51.00).

翻译：长上下文模型在处理长序列方面展现出巨大潜力，促进了众多实际应用的发展。其成功可归因于模型能够定位上下文中的隐含关键信息以进行后续预测。然而，近期研究表明，长上下文模型常易受上下文噪声（即无关标记）的影响，这些噪声可能误导模型的注意力机制。本文对上下文噪声进行了细粒度分析，并提出一种有效度量指标——积分梯度分数，用于检测和量化上下文中的噪声信息。研究发现，即使对检测到的上下文噪声进行简单缓解，也能显著增强模型对关键标记的注意力，并提升后续预测性能。基于此洞见，我们提出上下文去噪训练，这是一种简洁而有效的训练策略，旨在提升对关键标记的注意力，同时强化其对模型预测的影响。在上下文窗口扩展和长上下文对齐两种设定下，跨四个任务的广泛实验验证了上下文去噪训练的优越性。值得注意的是，采用上下文去噪训练后，一个开源的80亿参数模型能达到与GPT-4o（51.00分）相当的性能（50.92分）。