Large Language Models (LLMs) have demonstrated remarkable performance on various quantitative reasoning and knowledge benchmarks. However, many of these benchmarks are losing utility as LLMs get increasingly high scores, despite not yet reaching expert performance in these domains. We introduce ARB, a novel benchmark composed of advanced reasoning problems in multiple fields. ARB presents a more challenging test than prior benchmarks, featuring problems in mathematics, physics, biology, chemistry, and law. As a subset of ARB, we introduce a challenging set of math and physics problems which require advanced symbolic reasoning and domain knowledge. We evaluate recent models such as GPT-4 and Claude on ARB and demonstrate that current models score well below 50% on more demanding tasks. In order to improve both automatic and assisted evaluation capabilities, we introduce a rubric-based evaluation approach, allowing GPT-4 to score its own intermediate reasoning steps. Further, we conduct a human evaluation of the symbolic subset of ARB, finding promising agreement between annotators and GPT-4 rubric evaluation scores.

翻译：大型语言模型（LLMs）在各种定量推理和知识基准测试中展现出了卓越的性能。然而，随着LLMs在这些基准测试中得分持续攀升（尽管在相关领域尚未达到专家水平），许多基准的效用正在下降。我们提出ARB，这是一个由多领域高级推理问题组成的新型基准测试。相较于先前基准，ARB提供了更具挑战性的测试，涵盖数学、物理、生物学、化学和法律领域的问题。作为ARB的子集，我们引入了一组需要高级符号推理和领域知识的数学与物理难题。我们评估了GPT-4和Claude等近期模型在ARB上的表现，结果表明当前模型在较难任务上的得分远低于50%。为提升自动评估与辅助评估能力，我们提出一种基于评分准则的评估方法，使GPT-4能够对其自身的中间推理步骤进行评分。此外，我们对ARB的符号推理子集进行了人工评估，发现标注者评分与GPT-4评分准则评估结果之间存在高度一致性。