We present SciArena, an open and collaborative platform for evaluating foundation models on scientific literature-grounded tasks. Unlike traditional benchmarks for scientific literature understanding and synthesis, SciArena engages the research community directly, following the Chatbot Arena evaluation approach of community voting on model comparisons. By leveraging collective intelligence, SciArena offers a community-driven evaluation of model performance on open-ended scientific tasks that demand literature-grounded, long-form responses. The platform currently supports 47 foundation models and has collected over 20,000 votes from human researchers across diverse scientific domains. Our analysis of the data collected so far confirms its high quality. We discuss the results and insights based on the model ranking leaderboard. To further promote research in building model-based automated evaluation systems for literature tasks, we release SciArena-Eval, a meta-evaluation benchmark based on collected preference data. It measures the accuracy of models in judging answer quality by comparing their pairwise assessments with human votes. Our experiments highlight the benchmark's challenges and emphasize the need for more reliable automated evaluation methods.

翻译：我们推出SciArena，一个面向科学文献任务的开源协作式基础模型评估平台。与传统科学文献理解与综合基准不同，SciArena遵循Chatbot Arena的社区投票式模型比较评估方法，直接让研究社区参与其中。通过利用集体智慧，SciArena为需要基于文献的长篇回答的开放式科学任务提供社区驱动的模型性能评估。该平台目前支持47个基础模型，并已收集来自不同科学领域研究人员的超过20,000张投票。我们对现有数据的分析证实了其高质量特性。我们基于模型排名榜单讨论了相关结果与启示。为推进文献任务中基于模型的自动化评估系统研究，我们发布了基于收集偏好数据的元评估基准SciArena-Eval。该基准通过对比模型成对评估结果与人类投票数据，衡量模型在判断答案质量方面的准确性。我们的实验揭示了该基准面临的挑战，并强调了对更可靠自动化评估方法的迫切需求。