In this study, we aim to initiate the development of Radiology Foundation Model, termed as RadFM.We consider the construction of foundational models from the perspectives of data, model design, and evaluation thoroughly. Our contribution can be concluded as follows: (i), we construct a large-scale Medical Multi-modal Dataset, MedMD, consisting of 16M 2D and 3D medical scans. To the best of our knowledge, this is the first multi-modal dataset containing 3D medical scans. (ii), We propose an architecture that enables visually conditioned generative pre-training, allowing for the integration of text input interleaved with 2D or 3D medical scans to generate response for diverse radiologic tasks. The model was initially pre-trained on MedMD and subsequently domain-specific fine-tuned on RadMD, a radiologic cleaned version of MedMD, containing 3M radiologic visual-language pairs. (iii), we propose a new evaluation benchmark that comprises five tasks, aiming to comprehensively assess the capability of foundation models in handling practical clinical problems. Our experimental results confirm that RadFM significantly outperforms existing multi-modal foundation models. The codes, data, and model checkpoint will all be made publicly available to promote further research and development in the field.

翻译：本研究旨在启动放射学基础模型（命名为RadFM）的开发。我们从数据、模型设计和评估三个维度全面探讨了基础模型的构建。我们的贡献总结如下：（i）构建了一个大规模医学多模态数据集MedMD，包含1600万张2D和3D医学扫描图像。据我们所知，这是首个包含3D医学扫描的多模态数据集。（ii）我们提出了一种支持视觉条件生成预训练的架构，能够整合文本输入与2D或3D医学扫描图像的交错顺序，为多种放射学任务生成响应。模型首先在MedMD上进行预训练，随后在RadMD（MedMD的放射学清洗版本，包含300万放射学视觉-语言对）上进行领域特定微调。（iii）我们提出了一个新的评估基准，包含五项任务，旨在全面评估基础模型处理实际临床问题的能力。实验结果表明，RadFM显著优于现有的多模态基础模型。代码、数据和模型检查点将全部公开，以促进该领域的进一步研究与发展。