Event cameras sense the intensity changes asynchronously and produce event streams with high dynamic range and low latency. This has inspired research endeavors utilizing events to guide the challenging video superresolution (VSR) task. In this paper, we make the first attempt to address a novel problem of achieving VSR at random scales by taking advantages of the high temporal resolution property of events. This is hampered by the difficulties of representing the spatial-temporal information of events when guiding VSR. To this end, we propose a novel framework that incorporates the spatial-temporal interpolation of events to VSR in a unified framework. Our key idea is to learn implicit neural representations from queried spatial-temporal coordinates and features from both RGB frames and events. Our method contains three parts. Specifically, the Spatial-Temporal Fusion (STF) module first learns the 3D features from events and RGB frames. Then, the Temporal Filter (TF) module unlocks more explicit motion information from the events near the queried timestamp and generates the 2D features. Lastly, the SpatialTemporal Implicit Representation (STIR) module recovers the SR frame in arbitrary resolutions from the outputs of these two modules. In addition, we collect a real-world dataset with spatially aligned events and RGB frames. Extensive experiments show that our method significantly surpasses the prior-arts and achieves VSR with random scales, e.g., 6.5. Code and dataset are available at https: //vlis2022.github.io/cvpr23/egvsr.

翻译：事件相机以异步方式感知强度变化，生成具有高动态范围和低延迟的事件流。这激发了利用事件引导具有挑战性的视频超分辨率（VSR）任务的研究工作。本文首次尝试利用事件的高时间分辨率特性，解决在随机尺度下实现VSR的新问题。由于在引导VSR时难以表示事件的空间-时间信息，这一问题受到阻碍。为此，我们提出了一种新颖框架，将事件的空间-时间插值与VSR统一整合。核心思想是从查询的空间-时间坐标以及RGB帧和事件的特征中学习隐式神经表示。我们的方法包含三个部分：首先，时空融合（STF）模块从事件和RGB帧中学习3D特征；其次，时间滤波器（TF）模块从查询时间戳附近的事件中提取更显式的运动信息，并生成2D特征；最后，时空隐式表示（STIR）模块根据这两个模块的输出，恢复任意分辨率的超分辨率帧。此外，我们收集了一个真实世界数据集，包含空间对齐的事件和RGB帧。大量实验表明，我们的方法显著超越现有技术，并实现了随机尺度（例如6.5倍）的VSR。代码和数据集已发布于https://vlis2022.github.io/cvpr23/egvsr。