Language-supervised pre-training has proven to be a valuable method for extracting semantically meaningful features from images, serving as a foundational element in multimodal systems within the computer vision and medical imaging domains. However, resulting features are limited by the information contained within the text. This is particularly problematic in medical imaging, where radiologists' written findings focus on specific observations; a challenge compounded by the scarcity of paired imaging-text data due to concerns over leakage of personal health information. In this work, we fundamentally challenge the prevailing reliance on language supervision for learning general purpose biomedical imaging encoders. We introduce RAD-DINO, a biomedical image encoder pre-trained solely on unimodal biomedical imaging data that obtains similar or greater performance than state-of-the-art biomedical language supervised models on a diverse range of benchmarks. Specifically, the quality of learned representations is evaluated on standard imaging tasks (classification and semantic segmentation), and a vision-language alignment task (text report generation from images). To further demonstrate the drawback of language supervision, we show that features from RAD-DINO correlate with other medical records (e.g., sex or age) better than language-supervised models, which are generally not mentioned in radiology reports. Finally, we conduct a series of ablations determining the factors in RAD-DINO's performance; notably, we observe that RAD-DINO's downstream performance scales well with the quantity and diversity of training data, demonstrating that image-only supervision is a scalable approach for training a foundational biomedical image encoder.

翻译：语言监督预训练已被证明是从图像中提取语义有意义特征的有效方法，并成为计算机视觉和医学成像领域多模态系统的基础要素。然而，由此得到的特征受限于文本中包含的信息。这在医学成像中尤其成问题，因为放射科医生的书面发现聚焦于特定观察结果；同时，出于对个人健康信息泄露的担忧，配对成像-文本数据的稀缺性进一步加剧了这一挑战。在本研究中，我们从根本上挑战了当前依赖语言监督来学习通用生物医学图像编码器的范式。我们提出RAD-DINO，一种仅基于单模态生物医学成像数据预训练的 Biomedical 图像编码器，在多样化的基准测试中获得了与最先进的生物医学语言监督模型相似或更优的性能。具体而言，学习表征的质量通过标准成像任务（分类和语义分割）以及视觉-语言对齐任务（从图像生成文本报告）进行评估。为进一步证明语言监督的不足，我们展示了RAD-DINO的特征与放射报告中通常不提及的其他医疗记录（如性别或年龄）的相关性优于语言监督模型。最后，我们通过一系列消融实验确定了影响RAD-DINO性能的关键因素；值得注意的是，我们发现RAD-DINO的下游性能随训练数据的数量与多样性提升而良好扩展，这表明纯图像监督是训练基础生物医学图像编码器的一种可扩展方法。