Large Language Models (LLMs) have demonstrated remarkable performance on various quantitative reasoning and knowledge benchmarks. However, many of these benchmarks are losing utility as LLMs get increasingly high scores, despite not yet reaching expert performance in these domains. We introduce ARB, a novel benchmark composed of advanced reasoning problems in multiple fields. ARB presents a more challenging test than prior benchmarks, featuring problems in mathematics, physics, biology, chemistry, and law. As a subset of ARB, we introduce a challenging set of math and physics problems which require advanced symbolic reasoning and domain knowledge. We evaluate recent models such as GPT-4 and Claude on ARB and demonstrate that current models score well below 50% on more demanding tasks. In order to improve both automatic and assisted evaluation capabilities, we introduce a rubric-based evaluation approach, allowing GPT-4 to score its own intermediate reasoning steps. Further, we conduct a human evaluation of the symbolic subset of ARB, finding promising agreement between annotators and GPT-4 rubric evaluation scores.

翻译：大型语言模型在各种定量推理和知识基准测试中展现出卓越性能。然而，随着LLMs在这些测试中得分不断提高，许多基准测试的效用正在减弱，尽管它们在这些领域尚未达到专家水平。我们推出了ARB——一个由多个领域高级推理问题组成的新型基准测试。与以往的基准相比，ARB提出了更具挑战性的测试，涵盖数学、物理学、生物学、化学和法律等领域的问题。作为ARB的子集，我们引入了一组需要高级符号推理和领域知识的数学与物理难题。我们对GPT-4和Claude等近期模型在ARB上的表现进行了评估，结果显示当前模型在更具挑战性的任务上得分远低于50%。为提升自动评估与辅助评估能力，我们引入了一种基于评分标准的评估方法，使GPT-4能够对其自身的中间推理步骤进行评分。此外，我们对ARB的符号推理子集进行了人工评估，发现标注者与GPT-4的评分标准评估结果具有较高的一致性。