Recent accelerations in multi-modal applications have been made possible with the plethora of image and text data available online. However, the scarcity of analogous data in the medical field, specifically in histopathology, has halted comparable progress. To enable similar representation learning for histopathology, we turn to YouTube, an untapped resource of videos, offering $1,087$ hours of valuable educational histopathology videos from expert clinicians. From YouTube, we curate Quilt: a large-scale vision-language dataset consisting of $768,826$ image and text pairs. Quilt was automatically curated using a mixture of models, including large language models, handcrafted algorithms, human knowledge databases, and automatic speech recognition. In comparison, the most comprehensive datasets curated for histopathology amass only around $200$K samples. We combine Quilt with datasets from other sources, including Twitter, research papers, and the internet in general, to create an even larger dataset: Quilt-1M, with $1$M paired image-text samples, marking it as the largest vision-language histopathology dataset to date. We demonstrate the value of Quilt-1M by fine-tuning a pre-trained CLIP model. Our model outperforms state-of-the-art models on both zero-shot and linear probing tasks for classifying new histopathology images across $13$ diverse patch-level datasets of $8$ different sub-pathologies and cross-modal retrieval tasks.

翻译：摘要：近来多模态应用的加速发展得益于网络上丰富的图像与文本数据。然而，医学领域（尤其是组织病理学）中同类数据的匮乏阻碍了相应的进展。为促进组织病理学领域的表征学习，我们转向YouTube这一尚未被充分利用的视频资源，从中提取了总计1087小时由临床专家提供的优质教育性组织病理学视频。据此，我们构建了Quilt：一个大规模视觉语言数据集，包含768,826个图像-文本对。Quilt通过融合多种模型（包括大语言模型、人工算法、人类知识数据库及自动语音识别技术）自动完成数据整理。对比之下，现有最全面的组织病理学数据集仅积累约20万样本。我们将Quilt与其他来源（如推特、研究论文及互联网通用数据）的数据集合并，构建了规模更大的数据集Quilt-1M，其包含100万配对图像-文本样本，成为迄今最大的组织病理学视觉语言数据集。通过微调预训练的CLIP模型，我们验证了Quilt-1M的价值。该模型在零样本学习与线性探测任务中，对涵盖8种亚病理类型的13个不同斑块级数据集的新组织病理学图像进行分类，并在跨模态检索任务中均超越现有最优模型。