In this study, we aim to initiate the development of Radiology Foundation Model, termed as RadFM. We consider the construction of foundational models from three perspectives, namely, dataset construction, model design, and thorough evaluation. Our contribution can be concluded as follows: (i), we construct a large-scale Medical Multi-modal Dataset, MedMD, which consists of 16M 2D and 3D medical scans with high-quality text descriptions or reports across various data formats, modalities, and tasks, covering over 5000 distinct diseases. To the best of our knowledge, this is the first large-scale, high-quality, medical visual-language dataset, with both 2D and 3D scans; (ii), we propose an architecture that enables visually conditioned generative pre-training, i.e., allowing for integration of text input with 2D or 3D medical scans, and generate responses for diverse radiologic tasks. The model was initially pre-trained on MedMD and subsequently fine-tuned on the domain-specific dataset, which is a radiologic cleaned version of MedMD, containing 3M radiologic visual-language pairs, termed as RadMD; (iii), we propose a new evaluation benchmark, RadBench, that comprises five tasks, including modality recognition, disease diagnosis, visual question answering, report generation and rationale diagnosis, aiming to comprehensively assess the capability of foundation models in handling practical clinical problems. We conduct both automatic and human evaluation on RadBench, in both cases, RadFM outperforms existing multi-modal foundation models, that are publicaly accessible, including Openflamingo, MedFlamingo, MedVInT and GPT-4V. Additionally, we also adapt RadFM for different public benchmarks, surpassing existing SOTAs on diverse datasets. All codes, data, and model checkpoint will all be made publicly available to promote further research and development in the field.

翻译：摘要：本研究旨在启动放射学基础模型（RadFM）的开发。我们从三个维度考虑基础模型的构建：数据集构建、模型设计及全面评估。主要贡献如下：（i）构建大规模医学多模态数据集MedMD，包含1600万张2D与3D医学影像及对应的优质文本描述/报告，涵盖多种数据格式、模态与任务，覆盖超过5000种不同疾病。据我们所知，这是首个同时包含2D与3D影像的大规模高质量医学视觉语言数据集；（ii）提出一种支持视觉条件生成预训练的架构，即可将文本输入与2D或3D医学影像融合，为多种放射学任务生成响应。该模型先在MedMD上预训练，随后在领域特定数据集上微调——该数据集为MedMD经放射学清洗后的版本，包含300万放射学视觉语言对（命名为RadMD）；（iii）构建新评估基准RadBench，包含模态识别、疾病诊断、视觉问答、报告生成与推理诊断五项任务，旨在全面评估基础模型处理临床实际问题的能力。我们在RadBench上进行了自动评估与人工评估，结果显示RadFM均优于现有公开的多模态基础模型（包括Openflamingo、MedFlamingo、MedVInT及GPT-4V）。此外，我们还调整RadFM适配不同公开基准，在多个数据集上超越现有最佳方法。所有代码、数据及模型检查点将全部开源，以推动该领域的进一步研究与发展。