Large language models (LLMs) have accomplished remarkable reasoning performance in various domains. However, in the domain of reasoning tasks, we discover a frailty: LLMs are surprisingly brittle to the ordering of the premises, despite the fact that such ordering does not alter the underlying task. In particular, we observe that LLMs achieve the best performance when the premise order aligns with the context required in intermediate reasoning steps. For example, in deductive reasoning tasks, presenting the premises in the same order as the ground truth proof in the prompt (as opposed to random ordering) drastically increases the model's accuracy. We first examine the effect of premise ordering on deductive reasoning on a variety of LLMs, and our evaluation shows that permuting the premise order can cause a performance drop of over 30%. In addition, we release the benchmark R-GSM, based on GSM8K, to examine the ordering effect for mathematical problem-solving, and we again observe a significant drop in accuracy, relative to the original GSM8K benchmark.

翻译：大语言模型（LLMs）已在多个领域展现出卓越的推理能力。然而，在推理任务领域，我们发现一个脆弱性：尽管前提顺序并不改变底层任务本身，LLMs却对前提顺序表现出惊人的敏感性。具体而言，我们观察到当前提顺序与中间推理步骤所需的上下文一致时，LLMs能达到最佳性能。例如，在演绎推理任务中，按照提示中真实推理证明的顺序呈现前提（而非随机排序）可显著提升模型准确率。我们首先在多种LLMs上研究了前提顺序对演绎推理的影响，实验表明：打乱前提顺序可能导致性能下降超过30%。此外，基于GSM8K数据集，我们发布了R-GSM基准测试用于评估数学问题求解中的顺序效应，并再次观察到相比原始GSM8K基准测试，准确率出现显著下降。