Contemporary deep learning multi-scale deblurring models suffer from many issues: 1) They perform poorly on non-uniformly blurred images/videos; 2) Simply increasing the model depth with finer-scale levels cannot improve deblurring; 3) Individual RGB frames contain a limited motion information for deblurring; 4) Previous models have a limited robustness to spatial transformations and noise. Below, we extend the DMPHN model by several mechanisms to address the above issues: I) We present a novel self-supervised event-guided deep hierarchical Multi-patch Network (MPN) to deal with blurry images and videos via fine-to-coarse hierarchical localized representations; II) We propose a novel stacked pipeline, StackMPN, to improve the deblurring performance under the increased network depth; III) We propose an event-guided architecture to exploit motion cues contained in videos to tackle complex blur in videos; IV) We propose a novel self-supervised step to expose the model to random transformations (rotations, scale changes), and make it robust to Gaussian noises. Our MPN achieves the state of the art on the GoPro and VideoDeblur datasets with a 40x faster runtime compared to current multi-scale methods. With 30ms to process an image at 1280x720 resolution, it is the first real-time deep motion deblurring model for 720p images at 30fps. For StackMPN, we obtain significant improvements over 1.2dB on the GoPro dataset by increasing the network depth. Utilizing the event information and self-supervision further boost results to 33.83dB.

翻译：当代基于深度学习的多尺度去模糊模型存在诸多问题：1）对非均匀模糊图像/视频处理效果不佳；2）单纯通过更精细尺度增加模型深度无法提升去模糊性能；3）单个RGB帧包含的运动信息有限，难以有效去模糊；4）先前模型对空间变换和噪声的鲁棒性有限。本文通过多种机制扩展DMPHN模型以解决上述问题：I）提出一种新颖的自监督事件引导深度分层多块网络（MPN），通过由细到粗的分层局部化表征处理模糊图像与视频；II）提出新颖的堆叠流水线StackMPN，在增加网络深度的条件下提升去模糊性能；III）设计事件引导架构以提取视频中包含的运动线索，应对视频中的复杂模糊；IV）提出自监督训练步骤，使模型暴露于随机变换（旋转、尺度变化）中，增强对高斯噪声的鲁棒性。本模型在GoPro与VideoDeblur数据集上达到当前最优性能，运行速度相比现有多尺度方法提升40倍。在1280×720分辨率下处理单张图像仅需30毫秒，成为首个在30fps场景下实现720p图像实时处理的深度运动去模糊模型。通过增加网络深度，StackMPN在GoPro数据集上获得超过1.2dB的显著提升；进一步结合事件信息与自监督方法，峰值信噪比提升至33.83dB。