In this paper, we explore the potential of Snapshot Compressive Imaging (SCI) technique for recovering the underlying 3D scene representation from a single temporal compressed image. SCI is a cost-effective method that enables the recording of high-dimensional data, such as hyperspectral or temporal information, into a single image using low-cost 2D imaging sensors. To achieve this, a series of specially designed 2D masks are usually employed, which not only reduces storage requirements but also offers potential privacy protection. Inspired by this, to take one step further, our approach builds upon the powerful 3D scene representation capabilities of neural radiance fields (NeRF). Specifically, we formulate the physical imaging process of SCI as part of the training of NeRF, allowing us to exploit its impressive performance in capturing complex scene structures. To assess the effectiveness of our method, we conduct extensive evaluations using both synthetic data and real data captured by our SCI system. Extensive experimental results demonstrate that our proposed approach surpasses the state-of-the-art methods in terms of image reconstruction and novel view image synthesis. Moreover, our method also exhibits the ability to restore high frame-rate multi-view consistent images by leveraging SCI and the rendering capabilities of NeRF. The code is available at https://github.com/WU-CVGL/SCINeRF.

翻译：本文探索了快照压缩成像（SCI）技术从单张时间压缩图像中恢复潜在三维场景表示的潜力。SCI是一种经济高效的方法，能够使用低成本二维成像传感器将高维数据（如高光谱或时间信息）记录到单张图像中。为了实现这一目标，通常采用一系列专门设计的二维掩膜，这不仅降低了存储需求，还提供了潜在的隐私保护能力。受此启发，为更进一步推进研究，我们基于神经辐射场（NeRF）强大的三维场景表示能力构建了该方法。具体而言，我们将SCI的物理成像过程建模为NeRF训练的一部分，从而利用其在捕捉复杂场景结构方面的卓越性能。为评估方法的有效性，我们使用合成数据以及SCI系统捕获的真实数据进行了广泛评估。大量实验结果表明，所提方法在图像重建和新视角图像合成方面均超越了现有最优方法。此外，通过结合SCI与NeRF的渲染能力，该方法还能恢复高帧率的多视角一致性图像。代码开源地址为https://github.com/WU-CVGL/SCINeRF。