In this study, we aim to initiate the development of Radiology Foundation Model, termed as RadFM.We consider the construction of foundational models from the perspectives of data, model design, and evaluation thoroughly. Our contribution can be concluded as follows: (i), we construct a large-scale Medical Multi-modal Dataset, MedMD, consisting of 16M 2D and 3D medical scans. To the best of our knowledge, this is the first multi-modal dataset containing 3D medical scans. (ii), We propose an architecture that enables visually conditioned generative pre-training, allowing for the integration of text input interleaved with 2D or 3D medical scans to generate response for diverse radiologic tasks. The model was initially pre-trained on MedMD and subsequently domain-specific fine-tuned on RadMD, a radiologic cleaned version of MedMD, containing 3M radiologic visual-language pairs. (iii), we propose a new evaluation benchmark that comprises five tasks, aiming to comprehensively assess the capability of foundation models in handling practical clinical problems. Our experimental results confirm that RadFM significantly outperforms existing multi-modal foundation models. The codes, data, and model checkpoint will all be made publicly available to promote further research and development in the field.

翻译：在本研究中，我们旨在启动放射学基础模型（RadFM）的开发。我们从数据、模型设计与评估三个维度全面探讨了基础模型的构建。我们的贡献可总结如下：（i）构建了一个大规模医学多模态数据集MedMD，包含1600万张2D及3D医学扫描图像。据我们所知，这是首个涵盖3D医学扫描的多模态数据集。（ii）提出了一种支持视觉条件生成式预训练的架构，该架构能够整合穿插2D或3D医学扫描的文本输入，从而针对多样化的放射学任务生成响应。模型首先在MedMD上进行预训练，随后在RadMD（MedMD的放射学清洗版本，包含300万对放射学视觉-语言数据）上进行领域特异性微调。（iii）提出了一个包含五项任务的新型评估基准，旨在全面衡量基础模型处理实际临床问题的能力。实验结果证实，RadFM的性能显著优于现有的多模态基础模型。所有代码、数据及模型检查点将全部公开，以推动该领域的进一步研究与发展。