Neural Radiance Fields (NeRF) accomplishes photo-realistic novel view synthesis by learning the implicit volumetric representation of a scene from multi-view images, which faithfully convey the colorimetric information. However, sensor noises will contaminate low-value pixel signals, and the lossy camera image signal processor will further remove near-zero intensities in extremely dark situations, deteriorating the synthesis performance. Existing approaches reconstruct low-light scenes from raw images but struggle to recover texture and boundary details in dark regions. Additionally, they are unsuitable for high-speed models relying on explicit representations. To address these issues, we present Thermal-NeRF, which takes thermal and visible raw images as inputs, considering the thermal camera is robust to the illumination variation and raw images preserve any possible clues in the dark, to accomplish visible and thermal view synthesis simultaneously. Also, the first multi-view thermal and visible dataset (MVTV) is established to support the research on multimodal NeRF. Thermal-NeRF achieves the best trade-off between detail preservation and noise smoothing and provides better synthesis performance than previous work. Finally, we demonstrate that both modalities are beneficial to each other in 3D reconstruction.

翻译：神经辐射场（NeRF）通过从多视角图像中学习场景的隐式体积表示来实现照片级真实感的新视角合成，这些图像忠实地传达了色度信息。然而，传感器噪声会污染低值像素信号，而有损相机图像信号处理器会在极暗情况下进一步移除接近零的强度值，从而降低合成性能。现有方法从原始图像重建低光场景，但在暗区恢复纹理和边界细节方面存在困难。此外，它们不适用于依赖显式表示的高速模型。为了解决这些问题，我们提出了Thermal-NeRF，它以热成像和可见光原始图像为输入——考虑到热相机对光照变化具有鲁棒性，且原始图像保留了暗区中的任何潜在线索——同时实现可见光和热成像的新视角合成。同时，我们建立了首个多视角热成像与可见光数据集（MVTV），以支持多模态NeRF的研究。Thermal-NeRF在细节保留和噪声平滑之间实现了最佳权衡，并提供了优于先前工作的合成性能。最后，我们证明了这两种模态在三维重建中相互有益。