LLM-based reasoning models have enabled the development of agentic systems that act as co-scientists, assisting in multi-step scientific analysis. However, evaluating these systems is challenging, as it requires realistic, end-to-end research scenarios that integrate data analysis, interpretation, and the generation of new insights from the experimental data. To address this limitation, we introduce HeurekaBench, a framework to create benchmarks with exploratory, open-ended research questions for experimental datasets. Each such question is grounded in a scientific study and its corresponding code repository, and is created using a semi-automated pipeline that leverages multiple LLMs to extract insights and generate candidate workflows, which are then verified against reported findings. We instantiate the framework in single-cell biology to obtain sc-HeurekaBench benchmark and use it to compare state-of-the-art single-cell agents. We further showcase the benefits of our benchmark for quantitatively analyzing current design choices in agentic systems. We find that the addition of a critic module can improve ill-formed responses for open-source LLM-based agents by up to 22% and close the gap with their closed-source counterparts. Overall, HeurekaBench sets a path toward rigorous, end-to-end evaluation of scientific agents, grounding benchmark construction in real scientific workflows.

翻译：基于大语言模型的推理模型推动了作为协作者的智能体系统的发展，这些系统可协助进行多步骤的科学分析。然而，评估此类系统具有挑战性，因为它需要整合数据分析、解释以及从实验数据中生成新见解的、真实且端到端的研究场景。为应对这一局限，我们引入了HeurekaBench，这是一个为实验数据集创建包含探索性、开放式研究问题的基准测试框架。每个此类问题都基于一项科学研究及其对应的代码仓库，并通过一个半自动化流程创建，该流程利用多个大语言模型来提取见解并生成候选工作流，随后根据已报告的研究结果进行验证。我们在单细胞生物学领域实例化了该框架，获得了sc-HeurekaBench基准，并用其比较了最先进的单细胞分析智能体。我们进一步展示了本基准在定量分析当前智能体系统设计选择方面的优势。我们发现，添加一个评论模块可以将基于开源大语言模型的智能体对非规范问题的响应提升高达22%，并缩小其与闭源模型之间的差距。总体而言，HeurekaBench为科学智能体的严格、端到端评估开辟了道路，将基准构建建立在真实的科学工作流基础之上。