Colorectal cancer (CRC) micro-satellite instability (MSI) prediction on histopathology images is a challenging weakly supervised learning task that involves multi-instance learning on gigapixel images. To date, radiology images have proven to have CRC MSI information and efficient patient imaging techniques. Different data modalities integration offers the opportunity to increase the accuracy and robustness of MSI prediction. Despite the progress in representation learning from the whole slide images (WSI) and exploring the potential of making use of radiology data, CRC MSI prediction remains a challenge to fuse the information from multiple data modalities (e.g., pathology WSI and radiology CT image). In this paper, we propose $M^{2}$Fusion: a Bayesian-based multimodal multi-level fusion pipeline for CRC MSI. The proposed fusion model $M^{2}$Fusion is capable of discovering more novel patterns within and across modalities that are beneficial for predicting MSI than using a single modality alone, as well as other fusion methods. The contribution of the paper is three-fold: (1) $M^{2}$Fusion is the first pipeline of multi-level fusion on pathology WSI and 3D radiology CT image for MSI prediction; (2) CT images are the first time integrated into multimodal fusion for CRC MSI prediction; (3) feature-level fusion strategy is evaluated on both Transformer-based and CNN-based method. Our approach is validated on cross-validation of 352 cases and outperforms either feature-level (0.8177 vs. 0.7908) or decision-level fusion strategy (0.8177 vs. 0.7289) on AUC score.

翻译：结直肠癌（CRC）微卫星不稳定性（MSI）预测是一项具有挑战性的弱监督学习任务，涉及对千兆像素图像进行多实例学习。迄今为止，放射学图像已被证实包含CRC MSI信息及高效的患者成像技术。不同数据模态的整合为提高MSI预测的准确性和鲁棒性提供了机遇。尽管在全切片图像（WSI）表示学习及利用放射学数据潜力探索方面取得了进展，CRC MSI预测在多数据模态（如病理WSI和放射学CT图像）信息融合中仍面临挑战。本文提出$M^{2}$Fusion：一种基于贝叶斯的多模态多层级融合流程用于CRC MSI预测。所提出的融合模型$M^{2}$Fusion能够发现模态内部及跨模态的新模式，这些模式比单独使用单一模态或其他融合方法更有益于MSI预测。本文的贡献有三点：（1）$M^{2}$Fusion是首个针对病理WSI与3D放射学CT图像进行多层级融合的MSI预测流程；（2）首次将CT图像整合到CRC MSI预测的多模态融合中；（3）在基于Transformer和CNN的方法上均评估了特征级融合策略。该方法在352例交叉验证中得到了验证，其AUC得分优于特征级融合策略（0.8177 vs. 0.7908）和决策级融合策略（0.8177 vs. 0.7289）。