This paper focuses on Super-resolution for online video streaming data. Applying existing super-resolution methods to video streaming data is non-trivial for two reasons. First, to support application with constant interactions, video streaming has a high requirement for latency that most existing methods are less applicable, especially on low-end devices. Second, existing video streaming protocols (e.g., WebRTC) dynamically adapt the video quality to the network condition, thus video streaming in the wild varies greatly under different network bandwidths, which leads to diverse and dynamic degradations. To tackle the above two challenges, we proposed a novel video super-resolution method for online video streaming. First, we incorporate Look-Up Table (LUT) to lightweight convolution modules to achieve real-time latency. Second, for variant degradations, we propose a pixel-level LUT fusion strategy, where a set of LUT bases are built upon state-of-the-art SR networks pre-trained on different degraded data, and those LUT bases are combined with extracted weights from lightweight convolution modules to adaptively handle dynamic degradations. Extensive experiments are conducted on a newly proposed online video streaming dataset named LDV-WebRTC. All the results show that our method significantly outperforms existing LUT-based methods and offers competitive SR performance with faster speed compared to efficient CNN-based methods. Accelerated with our parallel LUT inference, our proposed method can even support online 720P video SR around 100 FPS.

翻译：本文聚焦于在线视频流数据的超分辨率问题。将现有超分辨率方法直接应用于视频流数据面临两大挑战。首先，为支持持续交互的应用场景，视频流对延迟具有极高要求，这使得多数现有方法难以适用，尤其是在低端设备上。其次，现有视频流协议（如WebRTC）会根据网络状况动态调整视频质量，因此实际场景中的视频流在不同网络带宽下差异显著，导致退化模式多样且动态变化。为应对上述挑战，我们提出了一种面向在线视频流的新型超分辨率方法。首先，我们引入查找表（LUT）以轻量化卷积模块，从而实现实时延迟。其次，针对变化的退化模式，我们提出像素级LUT融合策略：基于在不同退化数据上预训练的先进超分辨率网络构建一组LUT基元，并通过轻量化卷积模块提取的权重对这些基元进行组合，从而自适应处理动态退化。在最新提出的在线视频流数据集LDV-WebRTC上进行了大量实验。所有结果表明，我们的方法显著优于现有基于LUT的方法，并在保持更快速度的同时，实现了与高效CNN方法相媲美的超分辨率性能。通过并行LUT推理加速，我们提出的方法甚至可支持720P视频在线超分辨率处理，帧率约达100FPS。