Modern video generative models based on diffusion models can produce very realistic clips, but they are computationally inefficient, often requiring minutes of GPU time for just a few seconds of video. This inefficiency poses a critical barrier to deploying generative video in applications that require real-time interactions, such as embodied AI and VR/AR. This paper explores a new strategy for camera-conditioned video generation of static scenes: using diffusion-based generative models to generate a sparse set of keyframes, and then synthesizing the full video through 3D reconstruction and rendering. By lifting keyframes into a 3D representation and rendering intermediate views, our approach amortizes the generation cost across hundreds of frames while enforcing geometric consistency. We further introduce a model that predicts the optimal number of keyframes for a given camera trajectory, allowing the system to adaptively allocate computation. Our final method, SRENDER, uses very sparse keyframes for simple trajectories and denser ones for complex camera motion. This results in video generation that is more than 40 times faster than the diffusion-based baseline in generating 20 seconds of video, while maintaining high visual fidelity and temporal stability, offering a practical path toward efficient and controllable video synthesis.

翻译：基于扩散模型的现代视频生成模型能够生成非常逼真的视频片段，但计算效率低下，通常需要数分钟的GPU时间才能生成几秒钟的视频。这种低效性对在需要实时交互的应用（如具身人工智能和VR/AR）中部署生成式视频构成了关键障碍。本文探索了一种用于静态场景相机条件视频生成的新策略：使用基于扩散的生成模型生成稀疏的关键帧集合，然后通过三维重建和渲染合成完整视频。通过将关键帧提升为三维表示并渲染中间视图，我们的方法将生成成本分摊到数百帧中，同时确保了几何一致性。我们进一步引入了一个模型，用于预测给定相机轨迹的最佳关键帧数量，从而使系统能够自适应地分配计算资源。我们的最终方法SRENDER对简单轨迹使用非常稀疏的关键帧，对复杂的相机运动则使用更密集的关键帧。这使得在生成20秒视频时，其生成速度比基于扩散的基线方法快40倍以上，同时保持了高视觉保真度和时间稳定性，为高效可控的视频合成提供了一条实用路径。