Scene Dynamic Recovery (SDR) by inverting distorted Rolling Shutter (RS) images to an undistorted high frame-rate Global Shutter (GS) video is a severely ill-posed problem due to the missing temporal dynamic information in both RS intra-frame scanlines and inter-frame exposures, particularly when prior knowledge about camera/object motions is unavailable. Commonly used artificial assumptions on scenes/motions and data-specific characteristics are prone to producing sub-optimal solutions in real-world scenarios. To address this challenge, we propose an event-based SDR network within a self-supervised learning paradigm, i.e., SelfUnroll. We leverage the extremely high temporal resolution of event cameras to provide accurate inter/intra-frame dynamic information. Specifically, an Event-based Inter/intra-frame Compensator (E-IC) is proposed to predict the per-pixel dynamic between arbitrary time intervals, including the temporal transition and spatial translation. Exploring connections in terms of RS-RS, RS-GS, and GS-RS, we explicitly formulate mutual constraints with the proposed E-IC, resulting in supervisions without ground-truth GS images. Extensive evaluations over synthetic and real datasets demonstrate that the proposed method achieves state-of-the-art and shows remarkable performance for event-based RS2GS inversion in real-world scenarios. The dataset and code are available at https://w3un.github.io/selfunroll/.

翻译：场景动态恢复（SDR）旨在将畸变的卷帘快门（RS）图像反转为无畸变的高帧率全局快门（GS）视频，但由于RS帧内扫描线与帧间曝光均缺失时间动态信息，尤其是在未知相机/物体运动先验的情况下，该问题严重不适定。常见的人工假设场景/运动与数据特异性特征在真实场景中易产生次优解。为解决此挑战，我们提出一种基于事件的自监督学习范式下的SDR网络，即SelfUnroll。利用事件相机极高的时间分辨率来提供精确的帧间/帧内动态信息。具体而言，提出基于事件的帧间/帧内补偿器（E-IC），用于预测任意时间间隔内的逐像素动态，包括时间过渡与空间平移。通过探索RS-RS、RS-GS与GS-RS间的关联，利用所提E-IC显式构建相互约束，从而在无GS真值图像的情况下实现监督。在合成与真实数据集上的广泛评估表明，所提方法在真实场景下基于事件的RS2GS反演中达到最优性能。数据集与代码已公开于https://w3un.github.io/selfunroll/。