Video outpainting aims to adequately complete missing areas at the edges of video frames. Compared to image outpainting, it presents an additional challenge as the model should maintain the temporal consistency of the filled area. In this paper, we introduce a masked 3D diffusion model for video outpainting. We use the technique of mask modeling to train the 3D diffusion model. This allows us to use multiple guide frames to connect the results of multiple video clip inferences, thus ensuring temporal consistency and reducing jitter between adjacent frames. Meanwhile, we extract the global frames of the video as prompts and guide the model to obtain information other than the current video clip using cross-attention. We also introduce a hybrid coarse-to-fine inference pipeline to alleviate the artifact accumulation problem. The existing coarse-to-fine pipeline only uses the infilling strategy, which brings degradation because the time interval of the sparse frames is too large. Our pipeline benefits from bidirectional learning of the mask modeling and thus can employ a hybrid strategy of infilling and interpolation when generating sparse frames. Experiments show that our method achieves state-of-the-art results in video outpainting tasks. More results and codes are provided at our https://fanfanda.github.io/M3DDM/.

翻译：视频外延补全旨在充分填补视频帧边缘的缺失区域。与图像外延补全相比，其额外挑战在于模型需保持填充区域的时间一致性。本文提出一种掩码3D扩散模型用于视频外延补全。我们利用掩码建模技术训练3D扩散模型，从而可通过多个引导帧连接多个视频片段推理结果，确保时间一致性并减少相邻帧间的抖动。同时，我们提取视频全局帧作为提示，通过交叉注意力机制引导模型获取当前视频片段之外的全局信息。此外，我们引入一种混合的由粗到精推理流程，以缓解伪影累积问题。现有由粗到精流程仅采用填充策略，因稀疏帧时间间隔过大导致性能退化。而本流程得益于掩码建模的双向学习能力，在生成稀疏帧时可结合填充与插值的混合策略。实验表明，我们的方法在视频外延补全任务中达到了最先进水平。更多结果与代码已发布至：https://fanfanda.github.io/M3DDM/。