In this study, we aim to initiate the development of Radiology Foundation Model, termed as RadFM.We consider the construction of foundational models from the perspectives of dataset construction, model design, and thorough evaluation. Our contribution can be concluded as follows: (i), we construct a large-scale Medical Multi-modal Dataset, MedMD, which consists of 16M 2D and 3D medical scans with high-quality text descriptions or reports across various data formats, modalities, and tasks, covering over 5000 distinct diseases. To the best of our knowledge, this is the first large-scale, high-quality, medical visual-language dataset, with both 2D and 3D scans; (ii ), we propose an architecture that enables visually conditioned generative pre-training, i.e., allowing for integration of text input with 2D or 3D medical scans, and generate responses for diverse radiologic tasks. The model was initially pre-trained on MedMD and subsequently fine-tuned on the domain-specific dataset, which is a radiologic cleaned version of MedMD, containing 3M radiologic visual-language pairs, termed as RadMD; (iii), we propose a new evaluation benchmark, RadBench, that comprises five tasks, including modality recognition, disease diagnosis, visual question answering, report generation and rationale diagnosis, aiming to comprehensively assess the capability of foundation models in handling practical clinical problems. We conduct both automatic and human evaluation on RadBench, in both cases, RadFM significantly outperforms existing multi-modal foundation models. The codes, data, and model checkpoint will all be made publicly available to promote further research and development in the field.

翻译：在本研究中，我们旨在启动放射学基础模型（RadFM）的开发，从数据集构建、模型设计与全面评估三个维度探索基础模型的建设。我们的贡献可归纳如下：（i）构建了大规模医学多模态数据集MedMD，包含1600万份2D和3D医学扫描数据，附有跨多种数据格式、模态和任务的高质量文本描述或报告，覆盖5000余种疾病。据我们所知，这是首个涵盖2D与3D扫描的大规模高质量医学视觉语言数据集；（ii）提出一种支持视觉条件生成预训练的架构，可整合文本输入与2D/3D医学扫描，为多种放射学任务生成响应。模型首先在MedMD上预训练，随后在领域专用数据集（即MedMD的放射学清洗版本，含300万放射学视觉语言对，称为RadMD）上微调；（iii）提出全新评估基准RadBench，包含模态识别、疾病诊断、视觉问答、报告生成及原理诊断五项任务，旨在全面评估基础模型处理实际临床问题的能力。我们在RadBench上进行了自动与人工评估，两种情况下RadFM均显著优于现有多模态基础模型。相关代码、数据和模型检查点将全部公开，以促进该领域的进一步研究与发展。