Scene Dynamic Recovery (SDR) by inverting distorted Rolling Shutter (RS) images to an undistorted high frame-rate Global Shutter (GS) video is a severely ill-posed problem, particularly when prior knowledge about camera/object motions is unavailable. Commonly used artificial assumptions on motion linearity and data-specific characteristics, regarding the temporal dynamics information embedded in the RS scanlines, are prone to producing sub-optimal solutions in real-world scenarios. To address this challenge, we propose an event-based RS2GS framework within a self-supervised learning paradigm that leverages the extremely high temporal resolution of event cameras to provide accurate inter/intra-frame information. % In this paper, we propose to leverage the event camera to provide inter/intra-frame information as the emitted events have an extremely high temporal resolution and learn an event-based RS2GS network within a self-supervised learning framework, where real-world events and RS images can be exploited to alleviate the performance degradation caused by the domain gap between the synthesized and real data. Specifically, an Event-based Inter/intra-frame Compensator (E-IC) is proposed to predict the per-pixel dynamic between arbitrary time intervals, including the temporal transition and spatial translation. Exploring connections in terms of RS-RS, RS-GS, and GS-RS, we explicitly formulate mutual constraints with the proposed E-IC, resulting in supervisions without ground-truth GS images. Extensive evaluations over synthetic and real datasets demonstrate that the proposed method achieves state-of-the-art and shows remarkable performance for event-based RS2GS inversion in real-world scenarios. The dataset and code are available at https://w3un.github.io/selfunroll/.

翻译：场景动态恢复（Scene Dynamic Recovery, SDR）旨在将畸变的滚动快门（Rolling Shutter, RS）图像反向重建为无畸变的高帧率全局快门（Global Shutter, GS）视频，这是一个严重病态问题，尤其在缺乏相机/物体运动先验知识时更为突出。传统方法通常对运动线性及数据特性进行人为假设，并依赖RS扫描线中嵌入的时序动态信息，但在真实场景中容易产生次优解。针对这一挑战，我们提出一种基于事件的自监督学习框架RS2GS，利用事件相机极高的时间分辨率提供精确的帧间/帧内信息。具体而言，我们设计了一个事件驱动的帧间/帧内补偿器（Event-based Inter/intra-frame Compensator, E-IC），用于预测任意时间间隔内的逐像素动态变化，包括时间转换与空间平移。通过探索RS-RS、RS-GS和GS-RS之间的关联，我们利用所提出的E-IC显式构建相互约束，从而在无真实GS图像监督的情况下实现自监督学习。在合成数据集与真实数据集上的广泛评估表明，该方法在基于事件的RS2GS反向重建任务中达到了最先进水平，并在真实场景中展现出卓越性能。数据集与代码已开源：https://w3un.github.io/selfunroll/。