Computational imaging methods increasingly rely on powerful generative diffusion models to tackle challenging image restoration tasks. In particular, state-of-the-art zero-shot image inverse solvers leverage distilled text-to-image latent diffusion models (LDMs) to achieve unprecedented accuracy and perceptual quality with high computational efficiency. However, extending these advances to high-definition video restoration remains a significant challenge, due to the need to recover fine spatial detail while capturing subtle temporal dependencies. Consequently, methods that naively apply image-based LDM priors on a frame-by-frame basis often result in temporally inconsistent reconstructions. We address this challenge by leveraging recent advances in Video Consistency Models (VCMs), which distill video latent diffusion models into fast generators that explicitly capture temporal causality. Building on this foundation, we propose LVTINO, the first zero-shot or plug-and-play inverse solver for high definition video restoration with priors encoded by VCMs. Our conditioning mechanism bypasses the need for automatic differentiation and achieves state-of-the-art video reconstruction quality with only a few neural function evaluations, while ensuring strong measurement consistency and smooth temporal transitions across frames. Extensive experiments on a diverse set of video inverse problems show significant perceptual improvements over current state-of-the-art methods that apply image LDMs frame by frame, establishing a new benchmark in both reconstruction fidelity and computational efficiency. The code is available on GitHub.
翻译:计算成像方法日益依赖强大的生成扩散模型来解决具有挑战性的图像复原任务。特别是,最先进的零样本图像逆求解器利用蒸馏后的文本到图像潜在扩散模型(LDMs),以高计算效率实现了前所未有的精度和感知质量。然而,将这些进展扩展到高清视频复原仍然是一个重大挑战,因为需要在恢复精细空间细节的同时捕捉微妙的时间依赖性。因此,那些简单地在逐帧基础上应用基于图像的LDM先验的方法通常会导致时间不一致的重建结果。我们通过利用视频一致性模型(VCMs)的最新进展来应对这一挑战,该模型将视频潜在扩散模型蒸馏为能够显式捕获时间因果关系的快速生成器。在此基础上,我们提出了LVTINO——首个基于VCMs编码先验的、用于高清视频复原的零样本即插即用逆求解器。我们的条件机制绕过了对自动微分的要求,仅需少量神经函数评估即可实现最先进的视频重建质量,同时确保强大的测量一致性和帧间平滑的时间过渡。在多种视频逆问题上的大量实验表明,与当前逐帧应用图像LDMs的最先进方法相比,我们的方法在感知质量上取得了显著提升,在重建保真度和计算效率两方面均确立了新的基准。相关代码已在GitHub上开源。