This paper makes the first attempt to tackle the challenging task of recovering arbitrary frame rate latent global shutter (GS) frames from two consecutive rolling shutter (RS) frames, guided by the novel event camera data. Although events possess high temporal resolution, beneficial for video frame interpolation (VFI), a hurdle in tackling this task is the lack of paired GS frames. Another challenge is that RS frames are susceptible to distortion when capturing moving objects. To this end, we propose a novel self-supervised framework that leverages events to guide RS frame correction and VFI in a unified framework. Our key idea is to estimate the displacement field (DF) non-linear dense 3D spatiotemporal information of all pixels during the exposure time, allowing for the reciprocal reconstruction between RS and GS frames as well as arbitrary frame rate VFI. Specifically, the displacement field estimation (DFE) module is proposed to estimate the spatiotemporal motion from events to correct the RS distortion and interpolate the GS frames in one step. We then combine the input RS frames and DF to learn a mapping for RS-to-GS frame interpolation. However, as the mapping is highly under-constrained, we couple it with an inverse mapping (i.e., GS-to-RS) and RS frame warping (i.e., RS-to-RS) for self-supervision. As there is a lack of labeled datasets for evaluation, we generate two synthetic datasets and collect a real-world dataset to train and test our method. Experimental results show that our method yields comparable or better performance with prior supervised methods.

翻译：本文首次尝试解决一个具有挑战性的任务：在新型事件相机数据的引导下，从两个连续的卷帘快门（RS）帧中恢复任意帧率的潜在全局快门（GS）帧。尽管事件具有高时间分辨率，有利于视频帧插值（VFI），但处理该任务的一个障碍是缺乏配对的GS帧。另一个挑战是卷帘快门帧在捕捉运动物体时容易产生畸变。为此，我们提出一种新颖的自监督框架，利用事件在统一框架中引导RS帧校正和VFI。我们的核心思想是估计曝光时间内所有像素的位移场（DF）非线性稠密三维时空信息，从而支持RS与GS帧之间的互逆重建以及任意帧率的VFI。具体而言，我们提出位移场估计（DFE）模块，通过事件估计时空运动，一步完成RS畸变校正和GS帧插值。然后，我们将输入RS帧与DF结合，学习RS到GS帧插值的映射。然而，由于该映射高度欠约束，我们将其与逆映射（即GS到RS）以及RS帧扭曲（即RS到RS）结合以实现自监督。由于缺乏标注数据集用于评估，我们生成了两个合成数据集，并收集了一个真实世界数据集来训练和测试我们的方法。实验结果表明，我们的方法能够达到与先前监督方法相当或更优的性能。