There is growing interest in hypothesis generation with large language models (LLMs). However, fundamental questions remain: what makes a good hypothesis, and how can we systematically evaluate methods for hypothesis generation? To address this, we introduce HypoBench, a novel benchmark designed to evaluate LLMs and hypothesis generation methods across multiple aspects, including practical utility, generalizability, and hypothesis discovery rate. HypoBench includes 7 real-world tasks and 5 synthetic tasks with 194 distinct datasets. We evaluate four state-of-the-art LLMs combined with six existing hypothesis-generation methods. Overall, our results suggest that existing methods are capable of discovering valid and novel patterns in the data. However, the results from synthetic datasets indicate that there is still significant room for improvement, as current hypothesis generation methods do not fully uncover all relevant or meaningful patterns. Specifically, in synthetic settings, as task difficulty increases, performance significantly drops, with best models and methods only recovering 38.8% of the ground-truth hypotheses. These findings highlight challenges in hypothesis generation and demonstrate that HypoBench serves as a valuable resource for improving AI systems designed to assist scientific discovery.

翻译：大型语言模型（LLM）在假设生成方面的应用日益受到关注。然而，一些基本问题仍未解决：何为优质假设？如何系统评估假设生成方法？为此，我们提出HypoBench——一个旨在从实用性、泛化能力及假设发现率等多维度评估LLM与假设生成方法的新型基准测试框架。HypoBench涵盖7项真实世界任务与5项合成任务，共包含194个独立数据集。我们评估了四种前沿LLM与六种现有假设生成方法的组合性能。总体而言，实验结果表明现有方法能够有效发现数据中具有新颖性的有效模式。然而，合成数据集的结果显示当前方法仍有显著改进空间，因其未能完全揭示所有相关或有意义的模式。具体而言，在合成任务中，随着任务难度增加，模型性能显著下降，最优模型与方法仅能恢复38.8%的真实假设。这些发现揭示了假设生成领域面临的挑战，同时证明HypoBench可作为提升辅助科学发现的人工智能系统的重要资源。