Neural Radiance Fields (NeRF) is a revolutionary approach for rendering scenes by sampling a single ray per pixel and it has demonstrated impressive capabilities in novel-view synthesis from static scene images. However, in practice, we usually need to recover NeRF from unconstrained image collections, which poses two challenges: 1) the images often have dynamic changes in appearance because of different capturing time and camera settings; 2) the images may contain transient objects such as humans and cars, leading to occlusion and ghosting artifacts. Conventional approaches seek to address these challenges by locally utilizing a single ray to synthesize a color of a pixel. In contrast, humans typically perceive appearance and objects by globally utilizing information across multiple pixels. To mimic the perception process of humans, in this paper, we propose Cross-Ray NeRF (CR-NeRF) that leverages interactive information across multiple rays to synthesize occlusion-free novel views with the same appearances as the images. Specifically, to model varying appearances, we first propose to represent multiple rays with a novel cross-ray feature and then recover the appearance by fusing global statistics, i.e., feature covariance of the rays and the image appearance. Moreover, to avoid occlusion introduced by transient objects, we propose a transient objects handler and introduce a grid sampling strategy for masking out the transient objects. We theoretically find that leveraging correlation across multiple rays promotes capturing more global information. Moreover, extensive experimental results on large real-world datasets verify the effectiveness of CR-NeRF.

翻译：神经辐射场（NeRF）通过每像素单射线采样实现场景渲染，在静态场景图像的新视角合成中展现了卓越能力。然而实际应用中常需从无约束图像集合重建NeRF，这面临两大挑战：1）因拍摄时间与相机设置不同，图像外观存在动态变化；2）图像可能包含行人、车辆等瞬态物体，导致遮挡与鬼影伪影。传统方法通过局部利用单射线合成像素颜色来应对这些挑战，而人类通常借助多像素全局信息感知外观与物体。为模拟人类感知过程，本文提出跨射线NeRF（CR-NeRF），通过多射线交互信息合成无遮挡且与参考图像外观一致的新视角。具体而言，为建模动态外观，我们首先提出用新型跨射线特征表征多条射线，随后通过融合全局统计量（即射线特征协方差与图像外观）恢复外观。此外，为消除瞬态物体引起的遮挡，我们设计了瞬态物体处理器，并引入网格采样策略实现瞬态物体掩码。理论分析表明，利用多射线间相关性可促进全局信息捕获。在大型真实世界数据集上的大量实验结果验证了CR-NeRF的有效性。