Online video streaming has fundamental limitations on the transmission bandwidth and computational capacity and super-resolution is a promising potential solution. However, applying existing video super-resolution methods to online streaming is non-trivial. Existing video codecs and streaming protocols (\eg, WebRTC) dynamically change the video quality both spatially and temporally, which leads to diverse and dynamic degradations. Furthermore, online streaming has a strict requirement for latency that most existing methods are less applicable. As a result, this paper focuses on the rarely exploited problem setting of online streaming video super resolution. To facilitate the research on this problem, a new benchmark dataset named LDV-WebRTC is constructed based on a real-world online streaming system. Leveraging the new benchmark dataset, we proposed a novel method specifically for online video streaming, which contains a convolution and Look-Up Table (LUT) hybrid model to achieve better performance-latency trade-off. To tackle the changing degradations, we propose a mixture-of-expert-LUT module, where a set of LUT specialized in different degradations are built and adaptively combined to handle different degradations. Experiments show our method achieves 720P video SR around 100 FPS, while significantly outperforms existing LUT-based methods and offers competitive performance compared to efficient CNN-based methods.

翻译：在线视频流在传输带宽和计算能力上存在根本性限制，超分辨率是一种有潜力的解决方案。然而，将现有视频超分辨率方法应用于在线流媒体并非易事。现有视频编解码器和流媒体协议（如WebRTC）会在空间和时间上动态改变视频质量，导致退化模式的多样性和动态性。此外，在线流媒体对延迟有严格限制，而大多数现有方法难以满足这一要求。因此，本文聚焦于在线流媒体视频超分辨率这一鲜少被探索的问题设定。为促进该问题的研究，基于真实在线流媒体系统构建了名为LDV-WebRTC的新基准数据集。借助该新基准数据集，我们提出了一种专用于在线视频流的新方法，其中包含卷积与查找表（LUT）混合模型，以实现更优的性能-延迟权衡。为应对变化的退化模式，我们提出了专家混合LUT模块，该模块构建了一组专门处理不同退化模式的LUT，并通过自适应组合来处理各种退化。实验表明，我们的方法能以约100 FPS的速度实现720P视频超分辨率，显著优于现有基于LUT的方法，并与高效的基于CNN的方法性能相当。