Game engines are powerful tools in computer graphics. Their power comes at the immense cost of their development. In this work, we present a framework to train game-engine-like neural models, solely from monocular annotated videos. The result-a Learnable Game Engine (LGE)-maintains states of the scene, objects and agents in it, and enables rendering the environment from a controllable viewpoint. Similarly to a game engine, it models the logic of the game and the underlying rules of physics, to make it possible for a user to play the game by specifying both high- and low-level action sequences. Most captivatingly, our LGE unlocks the director's mode, where the game is played by plotting behind the scenes, specifying high-level actions and goals for the agents in the form of language and desired states. This requires learning "game AI", encapsulated by our animation model, to navigate the scene using high-level constraints, play against an adversary, devise the strategy to win a point. The key to learning such game AI is the exploitation of a large and diverse text corpus, collected in this work, describing detailed actions in a game and used to train our animation model. To render the resulting state of the environment and its agents, we use a compositional NeRF representation used in our synthesis model. To foster future research, we present newly collected, annotated and calibrated large-scale Tennis and Minecraft datasets. Our method significantly outperforms existing neural video game simulators in terms of rendering quality. Besides, our LGEs unlock applications beyond capabilities of the current state of the art. Our framework, data, and models are available at https://learnable-game-engines.github.io/lge-website.

翻译：游戏引擎是计算机图形学中的强大工具，其强大功能以巨大的开发成本为代价。本文提出一个框架，仅从单目标注视频中训练类游戏引擎的神经模型。其成果——可学习游戏引擎（LGE）——能够维护场景、物体及其中智能体的状态，并支持从可控视角渲染环境。与游戏引擎类似，它建模游戏逻辑与基础物理规则，使用户能够通过指定高层与低层动作序列来玩游戏。最引人注目的是，我们的LGE解锁了“导演模式”，即通过幕后布局来玩游戏——以语言和期望状态的形式为智能体指定高层动作与目标。这需要学习“游戏AI”，由我们的动画模型封装，利用高层约束导航场景、对抗对手、制定得分策略。学习此类游戏AI的关键在于利用一个大规模多样化的文本语料库（本文收集），该语料库描述了游戏中的详细动作，并用于训练动画模型。为渲染环境及其智能体的最终状态，我们采用合成模型中使用的组合式NeRF表征。为促进未来研究，我们发布了新收集、标注并校准的大规模网球与Minecraft数据集。本方法在渲染质量上显著优于现有神经视频游戏模拟器。此外，我们的LGE还解锁了超越当前技术水平的新应用。框架、数据及模型已开源至https://learnable-game-engines.github.io/lge-website。