Event cameras sense the intensity changes asynchronously and produce event streams with high dynamic range and low latency. This has inspired research endeavors utilizing events to guide the challenging video superresolution (VSR) task. In this paper, we make the first attempt to address a novel problem of achieving VSR at random scales by taking advantages of the high temporal resolution property of events. This is hampered by the difficulties of representing the spatial-temporal information of events when guiding VSR. To this end, we propose a novel framework that incorporates the spatial-temporal interpolation of events to VSR in a unified framework. Our key idea is to learn implicit neural representations from queried spatial-temporal coordinates and features from both RGB frames and events. Our method contains three parts. Specifically, the Spatial-Temporal Fusion (STF) module first learns the 3D features from events and RGB frames. Then, the Temporal Filter (TF) module unlocks more explicit motion information from the events near the queried timestamp and generates the 2D features. Lastly, the SpatialTemporal Implicit Representation (STIR) module recovers the SR frame in arbitrary resolutions from the outputs of these two modules. In addition, we collect a real-world dataset with spatially aligned events and RGB frames. Extensive experiments show that our method significantly surpasses the prior-arts and achieves VSR with random scales, e.g., 6.5. Code and dataset are available at https: //vlis2022.github.io/cvpr23/egvsr.

翻译：事件相机异步感知强度变化，产生具有高动态范围和低延迟的事件流，这激发了利用事件引导具有挑战性的视频超分辨率（VSR）任务的研究。本文首次尝试利用事件的高时间分辨率特性，解决随机尺度下实现VSR这一新问题。由于在引导VSR时难以表示事件的空间-时间信息，这一问题面临阻碍。为此，我们提出一个新颖框架，将事件的空间-时间插值统一集成到VSR中。核心思想是从查询的空间-时间坐标以及RGB帧和事件的特征中学习隐式神经表示。我们的方法包含三部分：首先，空间-时间融合（STF）模块从事件和RGB帧中学习3D特征；其次，时间滤波（TF）模块从查询时间戳附近的事件中提取更显式的运动信息并生成2D特征；最后，空间-时间隐式表示（STIR）模块根据前两个模块的输出恢复任意分辨率的超分辨率帧。此外，我们收集了一个包含空间对齐事件和RGB帧的真实世界数据集。大量实验表明，我们的方法显著优于现有技术，并实现了随机尺度（如6.5倍）的VSR。代码和数据集见https://vlis2022.github.io/cvpr23/egvsr。