The existence of completely aligned and paired multi-modal neuroimaging data has proved its effectiveness in diagnosis of brain diseases. However, collecting the full set of well-aligned and paired data is expensive or even impractical, since the practical difficulties may include high cost, long time acquisition, image corruption, and privacy issues. A realistic solution is to explore either an unsupervised learning or a semi-supervised learning to synthesize the absent neuroimaging data. In this paper, we are the first one to comprehensively approach cross-modality neuroimage synthesis task from different perspectives, which include the level of the supervision (especially for weakly-supervised and unsupervised), loss function, evaluation metrics, the range of modality synthesis, datasets (aligned, private and public) and the synthesis-based downstream tasks. To begin with, we highlight several opening challenges for cross-modality neuroimage sysnthesis. Then we summarize the architecture of cross-modality synthesis under various of supervision level. In addition, we provide in-depth analysis of how cross-modality neuroimage synthesis can improve the performance of different downstream tasks. Finally, we re-evaluate the open challenges and point out the future directions for the remaining challenges. All resources are available at https://github.com/M-3LAB/awesome-multimodal-brain-image-systhesis

翻译：完全对齐且配对的多模态神经影像数据在脑疾病诊断中已被证明具有有效性。然而，获取完整的良好对齐且配对的数据集成本高昂甚至不切实际，因为实际困难可能包括高成本、长采集时间、图像损坏和隐私问题。一种切实可行的解决方案是探索无监督学习或半监督学习方法来合成缺失的神经影像数据。本文首次从不同角度全面探讨跨模态神经影像合成任务，包括监督程度（特别是弱监督和无监督）、损失函数、评估指标、模态合成范围、数据集（对齐、私有和公开）以及基于合成的下游任务。首先，我们重点阐述了跨模态神经影像合成面临的若干开放性挑战。随后，我们归纳了不同监督水平下的跨模态合成架构。此外，我们深入分析了跨模态神经影像合成如何提升不同下游任务的性能。最后，我们重新评估了现有挑战并指出了未来研究方向。所有资源均可在 https://github.com/M-3LAB/awesome-multimodal-brain-image-systhesis 获取。