3D Gaussian Splatting-based techniques have recently advanced 3D scene reconstruction and novel view synthesis, achieving high-quality real-time rendering. However, these approaches are inherently limited by the underlying pinhole camera assumption in modeling the images and hence only work for All-in-Focus (AiF) sharp image inputs. This severely affects their applicability in real-world scenarios where images often exhibit defocus blur due to the limited depth-of-field (DOF) of imaging devices. Additionally, existing 3D Gaussian Splatting (3DGS) methods also do not support rendering of DOF effects. To address these challenges, we introduce DOF-GS that allows for rendering adjustable DOF effects, removing defocus blur as well as refocusing of 3D scenes, all from multi-view images degraded by defocus blur. To this end, we re-imagine the traditional Gaussian Splatting pipeline by employing a finite aperture camera model coupled with explicit, differentiable defocus rendering guided by the Circle-of-Confusion (CoC). The proposed framework provides for dynamic adjustment of DOF effects by changing the aperture and focal distance of the underlying camera model on-demand. It also enables rendering varying DOF effects of 3D scenes post-optimization, and generating AiF images from defocused training images. Furthermore, we devise a joint optimization strategy to further enhance details in the reconstructed scenes by jointly optimizing rendered defocused and AiF images. Our experimental results indicate that DOF-GS produces high-quality sharp all-in-focus renderings conditioned on inputs compromised by defocus blur, with the training process incurring only a modest increase in GPU memory consumption. We further demonstrate the applications of the proposed method for adjustable defocus rendering and refocusing of the 3D scene from input images degraded by defocus blur.

翻译：基于三维高斯溅射的技术近期推动了三维场景重建与新视角合成的发展，实现了高质量的实时渲染。然而，这些方法本质上受限于建模图像时所采用的针孔相机假设，因此仅适用于全聚焦清晰图像输入。这严重影响了其在现实场景中的适用性，因为实际图像常因成像设备有限的景深而呈现离焦模糊。此外，现有的三维高斯溅射方法亦不支持景深效果的渲染。为应对这些挑战，我们提出了DOF-GS，它能够从受离焦模糊退化的多视角图像中，实现可调景深效果渲染、离焦模糊消除以及三维场景重聚焦。为此，我们重新设计了传统高斯溅射流程，采用有限孔径相机模型，并结合由弥散圆引导的显式可微分离焦渲染。所提框架通过按需改变底层相机模型的光圈和焦距，实现了景深效果的动态调节。它还能在优化后渲染三维场景的不同景深效果，并从离焦训练图像生成全聚焦图像。此外，我们设计了一种联合优化策略，通过同步优化渲染的离焦图像与全聚焦图像，进一步提升重建场景的细节表现。实验结果表明，DOF-GS能够在输入图像受离焦模糊影响的情况下，生成高质量的全聚焦清晰渲染结果，且训练过程仅带来适度的GPU内存消耗增长。我们进一步展示了该方法在可调离焦渲染以及从离焦模糊退化输入图像实现三维场景重聚焦方面的应用潜力。