Automated analysis of chest radiography using deep learning has tremendous potential to enhance the clinical diagnosis of diseases in patients. However, deep learning models typically require large amounts of annotated data to achieve high performance -- often an obstacle to medical domain adaptation. In this paper, we build a data-efficient learning framework that utilizes radiology reports to improve medical image classification performance with limited labeled data (fewer than 1000 examples). Specifically, we examine image-captioning pretraining to learn high-quality medical image representations that train on fewer examples. Following joint pretraining of a convolutional encoder and transformer decoder, we transfer the learned encoder to various classification tasks. Averaged over 9 pathologies, we find that our model achieves higher classification performance than ImageNet-supervised and in-domain supervised pretraining when labeled training data is limited.

翻译：摘要：利用深度学习自动分析胸部X光影像在增强患者疾病临床诊断方面具有巨大潜力。然而，深度学习模型通常需要大量标注数据才能实现高性能——这往往是医学领域适应性应用的一大障碍。本文构建了一个数据高效学习框架，利用放射报告在有限标注数据（少于1000个样本）条件下提升医学影像分类性能。具体而言，我们研究了基于图像-描述联合预训练的方法，以学习高质量医学影像表征，从而减少训练样本需求。在对卷积编码器和Transformer解码器进行联合预训练后，我们将学到的编码器迁移至多种分类任务。在9种病理类型的平均表现中，我们发现当标注训练数据有限时，我们的模型比ImageNet监督预训练和领域内监督预训练实现了更高的分类性能。