Neural video game simulators emerged as powerful tools to generate and edit videos. Their idea is to represent games as the evolution of an environment's state driven by the actions of its agents. While such a paradigm enables users to play a game action-by-action, its rigidity precludes more semantic forms of control. To overcome this limitation, we augment game models with prompts specified as a set of natural language actions and desired states. The result-a Promptable Game Model (PGM)-makes it possible for a user to play the game by prompting it with high- and low-level action sequences. Most captivatingly, our PGM unlocks the director's mode, where the game is played by specifying goals for the agents in the form of a prompt. This requires learning "game AI", encapsulated by our animation model, to navigate the scene using high-level constraints, play against an adversary, and devise a strategy to win a point. To render the resulting state, we use a compositional NeRF representation encapsulated in our synthesis model. To foster future research, we present newly collected, annotated and calibrated Tennis and Minecraft datasets. Our method significantly outperforms existing neural video game simulators in terms of rendering quality and unlocks applications beyond the capabilities of the current state of the art. Our framework, data, and models are available at https://snap-research.github.io/promptable-game-models/.

翻译：神经视频游戏模拟器作为生成和编辑视频的强大工具崭露头角。其核心思想是将游戏表示为环境状态随智能体行动驱动的演化过程。尽管这一范式允许用户逐动作进行游戏，但其固化特性限制了更具语义性的操控方式。为突破这一局限，我们通过将自然语言动作和预期状态集作为提示来增强游戏模型。由此产生的可提示游戏模型（PGM）使用户能够通过高低层级动作序列的提示来操控游戏。更引人注目的是，我们的PGM解锁了导演模式——通过以提示形式为智能体设定目标来驱动游戏进程。这需要学习封装在动画模型中的"游戏AI"，使其能利用高层级约束导航场景、与对手博弈并制定得分策略。为渲染最终状态，我们采用合成模型中封装的组合式NeRF表征。为促进后续研究，我们发布了新采集、标注并校准的网球与《我的世界》数据集。本方法在渲染质量上显著超越现有神经视频游戏模拟器，并解锁了超越当前技术水平的应用场景。我们的框架、数据及模型已开源至https://snap-research.github.io/promptable-game-models/。