The advancements in artificial intelligence in recent years, such as Large Language Models (LLMs), have fueled expectations for breakthroughs in genomic foundation models (GFMs). The code of nature, hidden in diverse genomes since the very beginning of life's evolution, holds immense potential for impacting humans and ecosystems through genome modeling. Recent breakthroughs in GFMs, such as Evo, have attracted significant investment and attention to genomic modeling, as they address long-standing challenges and transform in-silico genomic studies into automated, reliable, and efficient paradigms. In the context of this flourishing era of consecutive technological revolutions in genomics, GFM studies face two major challenges: the lack of GFM benchmarking tools and the absence of open-source software for diverse genomics. These challenges hinder the rapid evolution of GFMs and their wide application in tasks such as understanding and synthesizing genomes, problems that have persisted for decades. To address these challenges, we introduce GFMBench, a framework dedicated to GFM-oriented benchmarking. GFMBench standardizes benchmark suites and automates benchmarking for a wide range of open-source GFMs. It integrates millions of genomic sequences across hundreds of genomic tasks from four large-scale benchmarks, democratizing GFMs for a wide range of in-silico genomic applications. Additionally, GFMBench is released as open-source software, offering user-friendly interfaces and diverse tutorials, applicable for AutoBench and complex tasks like RNA design and structure prediction. To facilitate further advancements in genome modeling, we have launched a public leaderboard showcasing the benchmark performance derived from AutoBench. GFMBench represents a step toward standardizing GFM benchmarking and democratizing GFM applications.

翻译：近年来人工智能的进步，特别是大语言模型（LLMs）的发展，激发了人们对基因组基础模型（GFMs）取得突破的期待。自生命演化之初便隐藏于多样基因组中的自然密码，通过基因组建模展现出影响人类与生态系统的巨大潜力。近期GFMs（如Evo模型）的重大突破，因其解决了长期存在的挑战并将计算基因组学研究转变为自动化、可靠且高效的范式，已吸引了大量投资与关注。在基因组学技术革命持续繁荣的背景下，GFM研究面临两大挑战：缺乏GFM基准测试工具以及缺少面向多样化基因组学的开源软件。这些挑战阻碍了GFMs的快速发展及其在理解与合成基因组等数十年悬而未决的任务中的广泛应用。为应对这些挑战，我们推出GFMBench——一个专为GFM导向的基准测试而设计的框架。GFMBench标准化了基准测试套件，并为广泛的开源GFMs提供自动化基准测试。它整合了来自四个大规模基准测试的数百项基因组任务中的数百万条基因组序列，使GFMs能够普惠于广泛的计算基因组学应用。此外，GFMBench以开源软件形式发布，提供用户友好的界面和多样化教程，适用于AutoBench及RNA设计与结构预测等复杂任务。为促进基因组建模的进一步发展，我们建立了公开排行榜，展示通过AutoBench获得的基准测试性能。GFMBench标志着向标准化GFM基准测试和普及GFM应用迈出的重要一步。