Online video super-resolution (online-VSR) highly relies on an effective alignment module to aggregate temporal information, while the strict latency requirement makes accurate and efficient alignment very challenging. Though much progress has been achieved, most of the existing online-VSR methods estimate the motion fields of each frame separately to perform alignment, which is computationally redundant and ignores the fact that the motion fields of adjacent frames are correlated. In this work, we propose an efficient Temporal Motion Propagation (TMP) method, which leverages the continuity of motion field to achieve fast pixel-level alignment among consecutive frames. Specifically, we first propagate the offsets from previous frames to the current frame, and then refine them in the neighborhood, which significantly reduces the matching space and speeds up the offset estimation process. Furthermore, to enhance the robustness of alignment, we perform spatial-wise weighting on the warped features, where the positions with more precise offsets are assigned higher importance. Experiments on benchmark datasets demonstrate that the proposed TMP method achieves leading online-VSR accuracy as well as inference speed. The source code of TMP can be found at \href{https://github.com/xtudbxk/TMP}{https://github.com/xtudbxk/TMP}.
翻译:在线视频超分辨率(online-VSR)高度依赖有效的对齐模块来聚合时序信息,而严格的延迟约束使得精准高效的对齐极具挑战性。尽管已取得诸多进展,现有在线VSR方法大多对每一帧独立估计运动场以完成对齐,这不仅存在计算冗余,更忽略了相邻帧运动场之间的关联性。本文提出一种高效的时序运动传播(TMP)方法,通过利用运动场的连续性实现连续帧间的快速像素级对齐。具体而言,我们首先将前一帧的偏移量传播至当前帧,随后在邻域内进行精细化修正,从而显著缩小匹配空间并加速偏移估计过程。此外,为增强对齐鲁棒性,我们对变形后的特征实施空间加权操作,为偏移更精确的位置赋予更高权重。基准数据集上的实验表明,所提出的TMP方法在在线VSR精度与推理速度方面均达到领先水平。TMP源代码可访问 \href{https://github.com/xtudbxk/TMP}{https://github.com/xtudbxk/TMP} 获取。