Existing deraining methods focus mainly on a single input image. However, with just a single input image, it is extremely difficult to accurately detect and remove rain streaks, in order to restore a rain-free image. In contrast, a light field image (LFI) embeds abundant 3D structure and texture information of the target scene by recording the direction and position of each incident ray via a plenoptic camera. LFIs are becoming popular in the computer vision and graphics communities. However, making full use of the abundant information available from LFIs, such as 2D array of sub-views and the disparity map of each sub-view, for effective rain removal is still a challenging problem. In this paper, we propose a novel method, 4D-MGP-SRRNet, for rain streak removal from LFIs. Our method takes as input all sub-views of a rainy LFI. To make full use of the LFI, it adopts 4D convolutional layers to simultaneously process all sub-views of the LFI. In the pipeline, the rain detection network, MGPDNet, with a novel Multi-scale Self-guided Gaussian Process (MSGP) module is proposed to detect high-resolution rain streaks from all sub-views of the input LFI at multi-scales. Semi-supervised learning is introduced for MSGP to accurately detect rain streaks by training on both virtual-world rainy LFIs and real-world rainy LFIs at multi-scales via computing pseudo ground truths for real-world rain streaks. We then feed all sub-views subtracting the predicted rain streaks into a 4D convolution-based Depth Estimation Residual Network (DERNet) to estimate the depth maps, which are later converted into fog maps. Finally, all sub-views concatenated with the corresponding rain streaks and fog maps are fed into a powerful rainy LFI restoring model based on the adversarial recurrent neural network to progressively eliminate rain streaks and recover the rain-free LFI.

翻译：现有去雨方法主要针对单张输入图像，然而仅凭单张图像，准确检测并去除雨痕以恢复无雨图像极为困难。相比之下，光场图像通过全光相机记录每束入射光线的方向与位置，嵌入了目标场景丰富的三维结构与纹理信息。光场图像在计算机视觉与图形学领域日益普及，但如何充分利用其多子视图二维阵列及每个子视图的视差图等丰富信息实现高效去雨，仍是挑战性难题。本文提出一种名为4D-MGP-SRRNet的新方法用于光场图像雨痕去除：该方法以含雨光场图像的所有子视图为输入，通过采用4D卷积层同时处理所有子视图以充分挖掘光场信息。在流程中，我们设计了包含新型多尺度自引导高斯过程模块的雨痕检测网络MGPDNet，用于从输入光场图像所有子视图中多尺度检测高分辨率雨痕。针对MSGP模块引入半监督学习，通过为真实雨痕计算伪真值，训练虚拟与现实场景多尺度含雨光场图像，实现精准雨痕检测。随后将各子视图减去预测雨痕后输入基于4D卷积的深度估计残差网络DERNet，估计深度图并进一步转化为雾图。最后，将各子视图与对应雨痕、雾图拼接后输入基于对抗循环神经网络的强力含雨光场图像修复模型，逐步消除雨痕并恢复无雨光场图像。