Unwanted camera occlusions, such as debris, dust, rain-drops, and snow, can severely degrade the performance of computer-vision systems. Dynamic occlusions are particularly challenging because of the continuously changing pattern. Existing occlusion-removal methods currently use synthetic aperture imaging or image inpainting. However, they face issues with dynamic occlusions as these require multiple viewpoints or user-generated masks to hallucinate the background intensity. We propose a novel approach to reconstruct the background from a single viewpoint in the presence of dynamic occlusions. Our solution relies for the first time on the combination of a traditional camera with an event camera. When an occlusion moves across a background image, it causes intensity changes that trigger events. These events provide additional information on the relative intensity changes between foreground and background at a high temporal resolution, enabling a truer reconstruction of the background content. We present the first large-scale dataset consisting of synchronized images and event sequences to evaluate our approach. We show that our method outperforms image inpainting methods by 3dB in terms of PSNR on our dataset.

翻译：动态遮挡物（如碎片、灰尘、雨滴和雪）会严重降低计算机视觉系统的性能。由于遮挡模式不断变化，动态遮挡问题尤为棘手。现有遮挡移除方法通常采用合成孔径成像或图像修复技术，但这些方法在处理动态遮挡时存在局限——它们需要多个视角或用户生成的掩膜来推测背景强度。我们提出了一种新颖方法，能够从单一视角重建存在动态遮挡时的背景。该方案首次将传统相机与事件相机相结合：当遮挡物在背景图像上移动时，由强度变化触发的事件能够提供前景与背景间相对强度变化的高时间分辨率信息，从而实现更真实的背景内容重建。我们发布了首个包含同步图像与事件序列的大规模数据集用于评估本方法，实验表明，该数据集上我们的方法在PSNR指标上比图像修复方法提升3dB。