We propose a novel Graph Neural Network-based method for segmentation based on data fusion of multimodal Scanning Electron Microscope (SEM) images. In most cases, Backscattered Electron (BSE) images obtained using SEM do not contain sufficient information for mineral segmentation. Therefore, imaging is often complemented with point-wise Energy-Dispersive X-ray Spectroscopy (EDS) spectral measurements that provide highly accurate information about the chemical composition but that are time-consuming to acquire. This motivates the use of sparse spectral data in conjunction with BSE images for mineral segmentation. The unstructured nature of the spectral data makes most traditional image fusion techniques unsuitable for BSE-EDS fusion. We propose using graph neural networks to fuse the two modalities and segment the mineral phases simultaneously. Our results demonstrate that providing EDS data for as few as 1% of BSE pixels produces accurate segmentation, enabling rapid analysis of mineral samples. The proposed data fusion pipeline is versatile and can be adapted to other domains that involve image data and point-wise measurements.

翻译：我们提出了一种新颖的基于图神经网络的方法，用于多模态扫描电子显微镜（SEM）图像的数据融合分割。在大多数情况下，使用SEM获取的背散射电子（BSE）图像不包含足够的信息用于矿物分割。因此，成像通常辅以点状能量色散X射线光谱（EDS）测量，该测量能提供高度准确的化学成分信息，但获取过程耗时。这促使我们将稀疏光谱数据与BSE图像结合用于矿物分割。光谱数据的非结构化特性使得大多数传统的图像融合技术不适用于BSE-EDS融合。我们提出使用图神经网络来融合这两种模态并同时分割矿物相。我们的结果表明，仅需为1%的BSE像素提供EDS数据即可实现准确分割，从而实现对矿物样品的快速分析。所提出的数据融合流程具有通用性，可适用于其他涉及图像数据和点状测量的领域。