In this paper, we propose a bi-modality medical image synthesis approach based on sequential generative adversarial network (GAN) and semi-supervised learning. Our approach consists of two generative modules that synthesize images of the two modalities in a sequential order. A method for measuring the synthesis complexity is proposed to automatically determine the synthesis order in our sequential GAN. Images of the modality with a lower complexity are synthesized first, and the counterparts with a higher complexity are generated later. Our sequential GAN is trained end-to-end in a semi-supervised manner. In supervised training, the joint distribution of bi-modality images are learned from real paired images of the two modalities by explicitly minimizing the reconstruction losses between the real and synthetic images. To avoid overfitting limited training images, in unsupervised training, the marginal distribution of each modality is learned based on unpaired images by minimizing the Wasserstein distance between the distributions of real and fake images. We comprehensively evaluate the proposed model using two synthesis tasks based on three types of evaluate metrics and user studies. Visual and quantitative results demonstrate the superiority of our method to the state-of-the-art methods, and reasonable visual quality and clinical significance. Code is made publicly available at https://github.com/hustlinyi/Multimodal-Medical-Image-Synthesis.

翻译：本文提出了一种基于序列生成对抗网络（GAN）与半监督学习的双模态医学图像合成方法。该方法包含两个生成模块，按序依次合成两种模态的图像。我们提出一种合成复杂度测量方法，用于自动确定序列GAN中的合成顺序：先合成复杂度较低的模态图像，再生成复杂度较高的对应图像。该序列GAN采用端到端的半监督训练方式。在监督训练中，通过显式最小化真实图像与合成图像之间的重构损失，从双模态配对图像中学习其联合分布。为避免有限训练样本导致的过拟合，在无监督训练中，基于非配对图像通过最小化真实图像与生成图像分布之间的Wasserstein距离，学习每种模态的边缘分布。我们基于三类评估指标和用户研究，在两个合成任务上全面评估了所提模型。视觉与定量结果表明，该方法优于现有最优技术，且具有合理的视觉质量与临床意义。代码已开源：https://github.com/hustlinyi/Multimodal-Medical-Image-Synthesis。