Existing works on video frame interpolation (VFI) mostly employ deep neural networks that are trained by minimizing the L1, L2, or deep feature space distance (e.g. VGG loss) between their outputs and ground-truth frames. However, recent works have shown that these metrics are poor indicators of perceptual VFI quality. Towards developing perceptually-oriented VFI methods, in this work we propose latent diffusion model-based VFI, LDMVFI. This approaches the VFI problem from a generative perspective by formulating it as a conditional generation problem. As the first effort to address VFI using latent diffusion models, we rigorously benchmark our method on common test sets used in the existing VFI literature. Our quantitative experiments and user study indicate that LDMVFI is able to interpolate video content with favorable perceptual quality compared to the state of the art, even in the high-resolution regime. Our code is available at https://github.com/danier97/LDMVFI.

翻译：现有的视频帧插值（VFI）研究工作大多采用深度神经网络，通过最小化输出与真实帧之间的L1、L2距离或深度特征空间距离（如VGG损失）进行训练。然而，近期研究表明这些指标难以有效反映VFI的感知质量。为开发面向感知的VFI方法，本文提出基于潜在扩散模型的LDMVFI方法。该方法从生成式视角将VFI问题建模为条件生成问题。作为首个利用潜在扩散模型解决VFI的研究，我们在现有VFI文献常用测试集上进行了严格基准测试。定量实验与用户研究表明，即使在高分辨率条件下，LDMVFI也能生成优于最先进方法的感知质量视频插值结果。我们的代码发布于https://github.com/danier97/LDMVFI。