Continuous space-time video super-resolution (C-STVSR) aims to simultaneously enhance video resolution and frame rate at an arbitrary scale. Recently, implicit neural representation (INR) has been applied to video restoration, representing videos as implicit fields that can be decoded at an arbitrary scale. However, the highly ill-posed nature of C-STVSR limits the effectiveness of current INR-based methods: they assume linear motion between frames and use interpolation or feature warping to generate features at arbitrary spatiotemporal positions with two consecutive frames. This restrains C-STVSR from capturing rapid and nonlinear motion and long-term dependencies (involving more than two frames) in complex dynamic scenes. In this paper, we propose a novel C-STVSR framework, called HR-INR, which captures both holistic dependencies and regional motions based on INR. It is assisted by an event camera, a novel sensor renowned for its high temporal resolution and low latency. To fully utilize the rich temporal information from events, we design a feature extraction consisting of (1) a regional event feature extractor - taking events as inputs via the proposed event temporal pyramid representation to capture the regional nonlinear motion and (2) a holistic event-frame feature extractor for long-term dependence and continuity motion. We then propose a novel INR-based decoder with spatiotemporal embeddings to capture long-term dependencies with a larger temporal perception field. We validate the effectiveness and generalization of our method on four datasets (both simulated and real data), showing the superiority of our method.

翻译：连续时空视频超分辨率（C-STVSR）旨在以任意尺度同时提升视频的空间分辨率和帧率。近年来，隐式神经表示（INR）被应用于视频复原任务，其将视频表示为可在任意尺度解码的隐式场。然而，C-STVSR的高度不适定性限制了现有基于INR方法的性能：这些方法假设帧间运动为线性，并利用插值或特征扭曲技术，仅通过连续两帧生成任意时空位置的特征。这导致C-STVSR在复杂动态场景中难以捕捉快速非线性运动以及涉及多帧（超过两帧）的长期依赖关系。本文提出一种新颖的C-STVSR框架——HR-INR，该框架基于INR同时捕捉整体依赖关系与局部运动，并借助以高时间分辨率和低延迟著称的新型传感器——事件相机进行辅助。为充分利用事件数据蕴含的丰富时间信息，我们设计了一种特征提取模块，包含：（1）局部事件特征提取器——通过提出的事件时间金字塔表示将事件作为输入，以捕捉局部非线性运动；（2）整体事件-帧特征提取器——用于建模长期依赖关系与连续性运动。随后，我们提出一种新颖的基于INR的解码器，结合时空嵌入来扩大时间感知范围，从而捕捉长期依赖。我们在四个数据集（包含仿真数据与真实数据）上验证了方法的有效性与泛化能力，结果证明了本方法的优越性。