In this study, we aim to initiate the development of Radiology Foundation Model, termed as RadFM.We consider the construction of foundational models from the perspectives of dataset construction, model design, and thorough evaluation. Our contribution can be concluded as follows: (i), we construct a large-scale Medical Multi-modal Dataset, MedMD, which consists of 16M 2D and 3D medical scans with high-quality text descriptions or reports across various data formats, modalities, and tasks, covering over 5000 distinct diseases. To the best of our knowledge, this is the first large-scale, high-quality, medical visual-language dataset, with both 2D and 3D scans; (ii ), we propose an architecture that enables visually conditioned generative pre-training, i.e., allowing for integration of text input with 2D or 3D medical scans, and generate responses for diverse radiologic tasks. The model was initially pre-trained on MedMD and subsequently fine-tuned on the domain-specific dataset, which is a radiologic cleaned version of MedMD, containing 3M radiologic visual-language pairs, termed as RadMD; (iii), we propose a new evaluation benchmark, RadBench, that comprises five tasks, including modality recognition, disease diagnosis, visual question answering, report generation and rationale diagnosis, aiming to comprehensively assess the capability of foundation models in handling practical clinical problems. We conduct both automatic and human evaluation on RadBench, in both cases, RadFM significantly outperforms existing multi-modal foundation models. The codes, data, and model checkpoint will all be made publicly available to promote further research and development in the field.

翻译：在本研究中，我们旨在启动放射学基础模型（RadFM）的开发工作。我们从数据集构建、模型设计与全面评估三个维度出发，探讨基础模型的构建方法。我们的贡献可总结如下：（i）构建大规模医学多模态数据集MedMD，包含1600万张二维与三维医学影像，并附有跨多种数据格式、模态及任务的高质量文本描述或报告，覆盖超过5000种不同疾病。据我们所知，这是首个同时包含二维和三维影像的大规模高质量医学视觉语言数据集；（ii）提出一种支持视觉条件生成式预训练的架构，即允许将文本输入与二维或三维医学影像进行整合，并为多样化的放射学任务生成响应。该模型首先在MedMD上进行预训练，随后在领域特定数据集（即MedMD的放射学清理版本，包含300万放射学视觉语言对，命名为RadMD）上进行微调；（iii）提出新的评估基准RadBench，涵盖模态识别、疾病诊断、视觉问答、报告生成及推理诊断五项任务，旨在全面评估基础模型处理实际临床问题的能力。我们在RadBench上开展了自动评估与人工评估，结果显示RadFM在两种评估方式下均显著优于现有多模态基础模型。相关代码、数据及模型参数将全部公开，以促进该领域的进一步研究与发展。