Most of the existing Large Language Model (LLM) benchmarks on scientific problem reasoning focus on problems grounded in high-school subjects and are confined to elementary algebraic operations. To systematically examine the reasoning capabilities required for solving complex scientific problems, we introduce an expansive benchmark suite SciBench for LLMs. SciBench contains a carefully curated dataset featuring a range of collegiate-level scientific problems from mathematics, chemistry, and physics domains. Based on the dataset, we conduct an in-depth benchmarking study of representative open-source and proprietary LLMs with various prompting strategies. The results reveal that the current LLMs fall short of delivering satisfactory performance, with the best overall score of merely 43.22%. Furthermore, through a detailed user study, we categorize the errors made by LLMs into ten problem-solving abilities. Our analysis indicates that no single prompting strategy significantly outperforms the others and some strategies that demonstrate improvements in certain problem-solving skills could result in declines in other skills. We envision that SciBench will catalyze further developments in the reasoning abilities of LLMs, thereby ultimately contributing to scientific research and discovery.

翻译：现有的大语言模型科学推理基准大多聚焦于高中难度的题目，且局限于初等代数运算。为系统考察求解复杂科学问题所需的推理能力，我们提出了一个针对大语言模型的扩展基准套件SciBench。该基准包含一个精心整理的数据集，涵盖数学、化学和物理领域的本科级科学问题。基于该数据集，我们采用多种提示策略，对具有代表性的开源和闭源大语言模型进行了深入的基准测试研究。结果表明，当前大语言模型未能达到令人满意的性能，最高综合得分仅为43.22%。此外，通过详细的用户研究，我们将大语言模型产生的错误归因于十种问题求解能力。分析表明，没有任何一种提示策略显著优于其他策略，且某些在特定解题技能上带来提升的策略可能导致其他技能的下降。我们期待SciBench能推动大语言模型推理能力的进一步发展，从而最终助力科学研究和发现。