Simultaneous functional PET/MR (sf-PET/MR) presents a cutting-edge multimodal neuroimaging technique. It provides an unprecedented opportunity for concurrently monitoring and integrating multifaceted brain networks built by spatiotemporally covaried metabolic activity, neural activity, and cerebral blood flow (perfusion). Albeit high scientific/clinical values, short in hardware accessibility of PET/MR hinders its applications, let alone modern AI-based PET/MR fusion models. Our objective is to develop a clinically feasible AI-based disease diagnosis model trained on comprehensive sf-PET/MR data with the power of, during inferencing, allowing single modality input (e.g., PET only) as well as enforcing multimodal-based accuracy. To this end, we propose MX-ARM, a multimodal MiXture-of-experts Alignment and Reconstruction Model. It is modality detachable and exchangeable, allocating different multi-layer perceptrons dynamically ("mixture of experts") through learnable weights to learn respective representations from different modalities. Such design will not sacrifice model performance in uni-modal situation. To fully exploit the inherent complex and nonlinear relation among modalities while producing fine-grained representations for uni-modal inference, we subsequently add a modal alignment module to line up a dominant modality (e.g., PET) with representations of auxiliary modalities (MR). We further adopt multimodal reconstruction to promote the quality of learned features. Experiments on precious multimodal sf-PET/MR data for Mild Cognitive Impairment diagnosis showcase the efficacy of our model toward clinically feasible precision medicine.

翻译：同步功能PET/MR（sf-PET/MR）是一种前沿的多模态神经影像技术，为同步监测和整合由时空协变代谢活动、神经活动及脑血流（灌注）构建的多面脑网络提供了前所未有的机遇。尽管具有极高的科学/临床价值，PET/MR硬件设备可及性的不足限制了其应用，更遑论基于现代人工智能的PET/MR融合模型。我们的目标是开发一种临床可行的基于人工智能的疾病诊断模型，该模型不仅能在训练过程中利用全面的sf-PET/MR数据，还能在推理阶段支持单模态输入（如仅PET）并强制执行基于多模态的准确性。为此，我们提出MX-ARM——一种多模态混合专家对齐与重建模型。该模型具备模态可拆卸与可交换特性，通过可学习权重动态分配不同的多层感知器（“混合专家”），从不同模态中学习各自的表征。这种设计不会在单模态情况下牺牲模型性能。为充分挖掘模态间固有的复杂非线性关系，同时为单模态推理生成细粒度表征，我们进一步引入模态对齐模块，将主导模态（如PET）与辅助模态（MR）的表征进行对齐。此外，我们采用多模态重建以提升学习特征的质量。在珍贵的多模态sf-PET/MR数据上进行轻度认知障碍诊断的实验，验证了本模型在实现临床可行精准医疗方面的有效性。