Accurate reconstruction of complex dynamic scenes from just a single viewpoint continues to be a challenging task in computer vision. Current dynamic novel view synthesis methods typically require videos from many different camera viewpoints, necessitating careful recording setups, and significantly restricting their utility in the wild as well as in terms of embodied AI applications. In this paper, we propose $\textbf{GCD}$, a controllable monocular dynamic view synthesis pipeline that leverages large-scale diffusion priors to, given a video of any scene, generate a synchronous video from any other chosen perspective, conditioned on a set of relative camera pose parameters. Our model does not require depth as input, and does not explicitly model 3D scene geometry, instead performing end-to-end video-to-video translation in order to achieve its goal efficiently. Despite being trained on synthetic multi-view video data only, zero-shot real-world generalization experiments show promising results in multiple domains, including robotics, object permanence, and driving environments. We believe our framework can potentially unlock powerful applications in rich dynamic scene understanding, perception for robotics, and interactive 3D video viewing experiences for virtual reality.

翻译：仅从单一视角精确重建复杂动态场景，在计算机视觉领域仍是一项具有挑战性的任务。当前的动态新视角合成方法通常需要从多个不同摄像机视角采集的视频，这不仅要求精心的录制设置，也极大地限制了其在自然场景及具身人工智能应用中的实用性。本文提出$\textbf{GCD}$——一种可控的单目动态视角合成框架，该框架利用大规模扩散先验，能够在给定任意场景视频的条件下，根据一组相对摄像机姿态参数，生成从任意其他选定视角同步拍摄的视频。我们的模型无需深度信息作为输入，也不显式建模三维场景几何，而是通过端到端的视频到视频转换来高效实现目标。尽管仅使用合成多视角视频数据进行训练，零样本真实世界泛化实验在机器人学、物体恒存性及驾驶环境等多个领域均显示出有前景的结果。我们相信该框架有望在丰富的动态场景理解、机器人感知以及虚拟现实的交互式三维视频观看体验中开启强大的应用前景。