Recently, breakthroughs in video modeling have allowed for controllable camera trajectories in generated videos. However, these methods cannot be directly applied to user-provided videos that are not generated by a video model. In this paper, we present ReCapture, a method for generating new videos with novel camera trajectories from a single user-provided video. Our method allows us to re-generate the reference video, with all its existing scene motion, from vastly different angles and with cinematic camera motion. Notably, using our method we can also plausibly hallucinate parts of the scene that were not observable in the reference video. Our method works by (1) generating a noisy anchor video with a new camera trajectory using multiview diffusion models or depth-based point cloud rendering and then (2) regenerating the anchor video into a clean and temporally consistent reangled video using our proposed masked video fine-tuning technique.

翻译：近年来，视频建模领域的突破使得生成视频中的摄像机轨迹可控成为可能。然而，这些方法无法直接应用于非视频模型生成的用户提供视频。本文提出ReCapture方法，能够从单一用户提供的视频生成具有新颖摄像机轨迹的新视频。该方法使我们能够以截然不同的视角并配合电影级摄像机运动，重新生成参考视频及其所有现有场景运动。值得注意的是，利用该方法我们还能合理推测参考视频中未观察到的场景部分。该方法通过以下步骤实现：(1) 使用多视角扩散模型或基于深度的点云渲染生成具有新摄像机轨迹的含噪锚点视频，随后(2) 通过我们提出的掩码视频微调技术将锚点视频重新生成为清晰且时间一致的重定向视频。