Diffusion models are just at a tipping point for image super-resolution task. Nevertheless, it is not trivial to capitalize on diffusion models for video super-resolution which necessitates not only the preservation of visual appearance from low-resolution to high-resolution videos, but also the temporal consistency across video frames. In this paper, we propose a novel approach, pursuing Spatial Adaptation and Temporal Coherence (SATeCo), for video super-resolution. SATeCo pivots on learning spatial-temporal guidance from low-resolution videos to calibrate both latent-space high-resolution video denoising and pixel-space video reconstruction. Technically, SATeCo freezes all the parameters of the pre-trained UNet and VAE, and only optimizes two deliberately-designed spatial feature adaptation (SFA) and temporal feature alignment (TFA) modules, in the decoder of UNet and VAE. SFA modulates frame features via adaptively estimating affine parameters for each pixel, guaranteeing pixel-wise guidance for high-resolution frame synthesis. TFA delves into feature interaction within a 3D local window (tubelet) through self-attention, and executes cross-attention between tubelet and its low-resolution counterpart to guide temporal feature alignment. Extensive experiments conducted on the REDS4 and Vid4 datasets demonstrate the effectiveness of our approach.

翻译：扩散模型正处于图像超分辨率任务的转折点。然而，将其应用于视频超分辨率并非易事，这要求不仅需要保持从低分辨率视频到高分辨率视频的视觉外观，还需确保帧间的时间一致性。本文提出了一种新方法——空间适应性与时间一致性（SATeCo），用于视频超分辨率。SATeCo的核心在于从低分辨率视频中学习时空引导，以校准潜在空间中的高分辨率视频去噪过程及像素空间中的视频重建。技术上，SATeCo冻结了预训练UNet和VAE的所有参数，仅优化UNet解码器与VAE中精心设计的两个模块：空间特征适应（SFA）模块和时间特征对齐（TFA）模块。SFA通过自适应估计每个像素的仿射参数来调制帧特征，为高分辨率帧合成提供逐像素引导；TFA则通过自注意力机制探索3D局部窗口（tubelet）内的特征交互，并执行tubelet与其低分辨率对应物之间的交叉注意力，以引导时间特征对齐。在REDS4和Vid4数据集上的大量实验验证了该方法的有效性。