We present MS-Splatting -- a multi-spectral 3D Gaussian Splatting (3DGS) framework that is able to generate multi-view consistent novel views from images of multiple, independent cameras with different spectral domains. In contrast to previous approaches, our method does not require cross-modal camera calibration and is versatile enough to model a variety of different spectra, including thermal and near-infra red, without any algorithmic changes. Unlike existing 3DGS-based frameworks that treat each modality separately (by optimizing per-channel spherical harmonics) and therefore fail to exploit the underlying spectral and spatial correlations, our method leverages a novel neural color representation that encodes multi-spectral information into a learned, compact, per-splat feature embedding. A shallow multi-layer perceptron (MLP) then decodes this embedding to obtain spectral color values, enabling joint learning of all bands within a unified representation. Our experiments show that this simple yet effective strategy is able to improve multi-spectral rendering quality, while also leading to improved per-spectra rendering quality over state-of-the-art methods. We demonstrate the effectiveness of this new technique in agricultural applications to render vegetation indices, such as normalized difference vegetation index (NDVI).

翻译：我们提出了MS-Splatting——一种多光谱三维高斯泼溅（3DGS）框架，能够从具有不同光谱域的多个独立相机图像中生成多视角一致的新视图。与先前方法相比，我们的方法不需要跨模态相机标定，并且足够灵活，能够建模包括热红外和近红外在内的多种不同光谱，而无需任何算法修改。与现有基于3DGS的框架（通过优化每通道球谐函数分别处理每个模态，因而未能利用潜在的光谱和空间相关性）不同，我们的方法采用了一种新颖的神经颜色表示，将多光谱信息编码为学习得到的紧凑的每泼溅特征嵌入。随后，一个浅层多层感知机（MLP）解码该嵌入以获得光谱颜色值，从而实现在统一表示中联合学习所有波段。我们的实验表明，这种简单而有效的策略能够提高多光谱渲染质量，同时在单光谱渲染质量上也优于现有最先进方法。我们在农业应用中展示了这项新技术在渲染植被指数（如归一化差异植被指数NDVI）方面的有效性。