We propose a method for text-driven perpetual view generation -- synthesizing long videos of arbitrary scenes solely from an input text describing the scene and camera poses. We introduce a novel framework that generates such videos in an online fashion by combining the generative power of a pre-trained text-to-image model with the geometric priors learned by a pre-trained monocular depth prediction model. To achieve 3D consistency, i.e., generating videos that depict geometrically-plausible scenes, we deploy an online test-time training to encourage the predicted depth map of the current frame to be geometrically consistent with the synthesized scene; the depth maps are used to construct a unified mesh representation of the scene, which is updated throughout the generation and is used for rendering. In contrast to previous works, which are applicable only for limited domains (e.g., landscapes), our framework generates diverse scenes, such as walkthroughs in spaceships, caves, or ice castles. Project page: https://scenescape.github.io/

翻译：我们提出了一种文本驱动的永恒视角生成方法——仅根据描述场景和相机姿态的输入文本，即可合成任意场景的长视频。我们引入了一个新颖框架，通过结合预训练文本到图像模型的生成能力与预训练单目深度预测模型学习的几何先验，以在线方式生成此类视频。为了实现3D一致性（即生成描绘几何合理场景的视频），我们采用在线测试时训练，促使当前帧的预测深度图与合成场景在几何上保持一致；深度图用于构建场景的统一网格表示，该表示在生成过程中持续更新并用于渲染。与先前仅适用于有限领域（如风景）的工作不同，我们的框架可生成多样化场景，例如太空船、洞穴或冰城堡的漫游。项目页面：https://scenescape.github.io/