Self-supervised representation learning has been highly promising for histopathology image analysis with numerous approaches leveraging their patient-slide-patch hierarchy to learn better representations. In this paper, we explore how the combination of domain specific natural language information with such hierarchical visual representations can benefit rich representation learning for medical image tasks. Building on automated language description generation for features visible in histopathology images, we present a novel language-tied self-supervised learning framework, Hierarchical Language-tied Self-Supervision (HLSS) for histopathology images. We explore contrastive objectives and granular language description based text alignment at multiple hierarchies to inject language modality information into the visual representations. Our resulting model achieves state-of-the-art performance on two medical imaging benchmarks, OpenSRH and TCGA datasets. Our framework also provides better interpretability with our language aligned representation space. Code is available at https://github.com/Hasindri/HLSS.

翻译：自监督表征学习在组织病理学图像分析领域展现出巨大潜力，众多方法利用患者-切片-图像块层级结构来学习更优质的表征。本文探究领域特异性自然语言信息与层级化视觉表征相结合，如何促进医学图像任务的丰富表征学习。基于组织病理学图像可见特征的自动化语言描述生成技术，我们提出了一种新型语言关联自监督学习框架——层级式语言关联自监督方法（HLSS），用于组织病理学图像分析。通过对比学习目标与多层级粒度语言描述文本对齐，我们将语言模态信息注入视觉表征。最终模型在OpenSRH和TCGA两个医学影像基准数据集上取得了最优性能。此外，语言对齐的表征空间为模型提供了更优的可解释性。代码发布在https://github.com/Hasindri/HLSS。