Achieving accurate material segmentation for 3-channel RGB images is challenging due to the considerable variation in a material's appearance. Hyperspectral images, which are sets of spectral measurements sampled at multiple wavelengths, theoretically offer distinct information for material identification, as variations in intensity of electromagnetic radiation reflected by a surface depend on the material composition of a scene. However, existing hyperspectral datasets are impoverished regarding the number of images and material categories for the dense material segmentation task, and collecting and annotating hyperspectral images with a spectral camera is prohibitively expensive. To address this, we propose a new model, the MatSpectNet to segment materials with recovered hyperspectral images from RGB images. The network leverages the principles of colour perception in modern cameras to constrain the reconstructed hyperspectral images and employs the domain adaptation method to generalise the hyperspectral reconstruction capability from a spectral recovery dataset to material segmentation datasets. The reconstructed hyperspectral images are further filtered using learned response curves and enhanced with human perception. The performance of MatSpectNet is evaluated on the LMD dataset as well as the OpenSurfaces dataset. Our experiments demonstrate that MatSpectNet attains a 1.60% increase in average pixel accuracy and a 3.42% improvement in mean class accuracy compared with the most recent publication. The project code is attached to the supplementary material and will be published on GitHub.

翻译：实现基于三通道RGB图像的精确材料分割极具挑战性，因为材料外观存在显著差异。高光谱图像作为多个波长采样点的光谱测量集合，理论上能为材料识别提供独特信息——场景中材料成分决定了表面反射电磁辐射强度的变化。然而，现有高光谱数据集在面向密集材料分割任务时存在图像数量与材料类别匮乏的问题，而使用光谱相机采集并标注高光谱图像的成本极其高昂。为解决此问题，我们提出新型模型MatSpectNet，通过从RGB图像恢复的高光谱图像实现材料分割。该网络利用现代相机颜色感知原理约束重建的高光谱图像，并采用领域自适应方法将高光谱重建能力从光谱恢复数据集泛化到材料分割数据集。重建的高光谱图像进一步通过学习响应曲线进行滤波，并结合人类感知增强。我们在LMD数据集与OpenSurfaces数据集上评估了MatSpectNet的性能。实验表明，与最新文献相比，MatSpectNet的平均像素精度提升1.60%，平均类别精度提升3.42%。项目代码已附于补充材料中，并将发布于GitHub。