The target of space-time video super-resolution (STVSR) is to increase both the frame rate (also referred to as the temporal resolution) and the spatial resolution of a given video. Recent approaches solve STVSR using end-to-end deep neural networks. A popular solution is to first increase the frame rate of the video; then perform feature refinement among different frame features; and last increase the spatial resolutions of these features. The temporal correlation among features of different frames is carefully exploited in this process. The spatial correlation among features of different (spatial) resolutions, despite being also very important, is however not emphasized. In this paper, we propose a spatial-temporal feature interaction network to enhance STVSR by exploiting both spatial and temporal correlations among features of different frames and spatial resolutions. Specifically, the spatial-temporal frame interpolation module is introduced to interpolate low- and high-resolution intermediate frame features simultaneously and interactively. The spatial-temporal local and global refinement modules are respectively deployed afterwards to exploit the spatial-temporal correlation among different features for their refinement. Finally, a novel motion consistency loss is employed to enhance the motion continuity among reconstructed frames. We conduct experiments on three standard benchmarks, Vid4, Vimeo-90K and Adobe240, and the results demonstrate that our method improves the state of the art methods by a considerable margin. Our codes will be available at https://github.com/yuezijie/STINet-Space-time-Video-Super-resolution.

翻译：时空视频超分辨率（STVSR）的目标是同时提升给定视频的帧率（也称为时间分辨率）和空间分辨率。现有方法通常采用端到端深度神经网络来解决STVSR问题，其中一种主流方案是：首先提升视频帧率，然后对不同帧的特征进行精细化处理，最后提升这些特征的空间分辨率。在这一过程中，不同帧特征之间的时间相关性得到了精细利用，然而不同（空间）分辨率特征之间的空间相关性尽管同样重要，却未得到充分强调。本文提出了一种时空特征交互网络，通过综合利用不同帧与不同空间分辨率特征间的时空相关性来增强STVSR性能。具体而言，我们引入时空帧插值模块，以交互方式同时生成低分辨率与高分辨率的中间帧特征。随后分别部署时空局部与全局精细化模块，利用不同特征间的时空相关性进行特征优化。最后，采用新型运动一致性损失函数来增强重建帧之间的运动连续性。在Vid4、Vimeo-90K和Adobe240三个标准基准数据集上的实验表明，我们的方法显著优于现有最先进技术。相关代码将发布于https://github.com/yuezijie/STINet-Space-time-Video-Super-resolution。