Recently, there has been growing interest in leveraging large language models (LLMs) to generate symbolic world models from textual descriptions. Although LLMs have been extensively explored in the context of world modeling, prior studies encountered several challenges, including evaluation randomness, dependence on indirect metrics, and a limited domain scope. To address these limitations, we introduce a novel benchmark, Text2World, based on planning domain definition language (PDDL), featuring hundreds of diverse domains and employing multi-criteria, execution-based metrics for a more robust evaluation. We benchmark current LLMs using Text2World and find that reasoning models trained with large-scale reinforcement learning outperform others. However, even the best-performing model still demonstrates limited capabilities in world modeling. Building on these insights, we examine several promising strategies to enhance the world modeling capabilities of LLMs, including test-time scaling, agent training, and more. We hope that Text2World can serve as a crucial resource, laying the groundwork for future research in leveraging LLMs as world models. The project page is available at https://text-to-world.github.io/.

翻译：近年来，利用大语言模型从文本描述生成符号化世界模型的研究兴趣日益增长。尽管LLMs在世界建模领域已得到广泛探索，但先前的研究面临若干挑战，包括评估随机性、对间接指标的依赖以及有限的领域范围。为应对这些局限，我们引入了一个基于规划领域定义语言的新型基准测试Text2World，该基准包含数百个多样化领域，并采用多准则、基于执行的指标以实现更稳健的评估。我们使用Text2World对当前主流LLMs进行基准测试，发现经过大规模强化学习训练的逻辑推理模型表现优于其他模型。然而，即使是性能最佳的模型，其世界建模能力仍显不足。基于这些发现，我们探讨了若干提升LLMs世界建模能力的潜在策略，包括测试时扩展、智能体训练等。我们希望Text2World能成为重要资源，为未来利用LLMs作为世界模型的研究奠定基础。项目页面详见 https://text-to-world.github.io/。