The target of space-time video super-resolution (STVSR) is to increase both the frame rate (also referred to as the temporal resolution) and the spatial resolution of a given video. Recent approaches solve STVSR using end-to-end deep neural networks. A popular solution is to first increase the frame rate of the video; then perform feature refinement among different frame features; and last increase the spatial resolutions of these features. The temporal correlation among features of different frames is carefully exploited in this process. The spatial correlation among features of different (spatial) resolutions, despite being also very important, is however not emphasized. In this paper, we propose a spatial-temporal feature interaction network to enhance STVSR by exploiting both spatial and temporal correlations among features of different frames and spatial resolutions. Specifically, the spatial-temporal frame interpolation module is introduced to interpolate low- and high-resolution intermediate frame features simultaneously and interactively. The spatial-temporal local and global refinement modules are respectively deployed afterwards to exploit the spatial-temporal correlation among different features for their refinement. Finally, a novel motion consistency loss is employed to enhance the motion continuity among reconstructed frames. We conduct experiments on three standard benchmarks, Vid4, Vimeo-90K and Adobe240, and the results demonstrate that our method improves the state of the art methods by a considerable margin. Our codes will be available at https://github.com/yuezijie/STINet-Space-time-Video-Super-resolution.

翻译：时空视频超分辨率（STVSR）的目标是同时提升给定视频的帧率（即时间分辨率）与空间分辨率。现有方法通过端到端深度神经网络实现STVSR，其中典型的解决方案是先提升视频帧率，随后对不同帧特征进行特征精炼，最后提升这些特征的空间分辨率。在此过程中，不同帧特征间的时间相关性得到了充分利用，而不同（空间）分辨率特征间的空间相关性虽同样重要，却未被充分强调。本文提出一种时空特征交互网络，通过同时挖掘不同帧与不同空间分辨率特征间的时空相关性来增强STVSR性能。具体而言，引入时空帧插值模块以交互式同步插值低分辨率与高分辨率的中间帧特征，随后分别部署时空局部与全局精炼模块，利用不同特征间的时空相关性进行特征优化，最后采用新颖的运动一致性损失函数增强重建帧间的运动连续性。在Vid4、Vimeo-90K与Adobe240三个标准基准数据集上的实验表明，本方法显著提升了现有最优方法的性能。相关代码将发布于https://github.com/yuezijie/STINet-Space-time-Video-Super-resolution。