Large Language Models (LLMs) tend to generate a long reasoning chain when solving complex tasks. However, as the reasoning chain extends, critical intermediate steps and the original prompt will be buried in the context, receiving insufficient attention and leading to errors. In this work, we present ATAR, a novel reasoning method that leverages the inherent reasoning structure to steer LLM attention. Our experiments show that ATAR outperforms SOTA methods across six benchmarks, achieving up to 15.39% absolute improvement. Furthermore, with ATAR, "non-reasoning" models achieve comparable or even better performance compared to reasoning models of the same size in most benchmarks. Finally, our ablation studies show that the attention alignment component contributes significantly, and that these improvements are persist under different attentionsteering backends.

翻译：大型语言模型在解决复杂任务时倾向于生成长推理链。然而，随着推理链的延伸，关键中间步骤与原始提示会被掩埋在上下文中，因注意力不足而导致错误。本文提出了一种名为ATAR的新型推理方法，该方法利用推理的内在结构引导大语言模型的注意力。实验表明，ATAR在六个基准测试上均优于现有最优方法，绝对性能提升最高达15.39%。此外，采用ATAR后，多数基准测试中"非推理型"模型的表现可与同等规模的"推理型"模型相当甚至更优。最后，消融实验证明注意力对齐组件贡献显著，且这些改进在不同注意力引导后端下均能保持稳定。