Histology analysis of the tumor micro-environment integrated with genomic assays is the gold standard for most cancers in modern medicine. This paper proposes a Gene-induced Multimodal Pre-training (GiMP) framework, which jointly incorporates genomics and Whole Slide Images (WSIs) for classification tasks. Our work aims at dealing with the main challenges of multi-modality image-omic classification w.r.t. (1) the patient-level feature extraction difficulties from gigapixel WSIs and tens of thousands of genes, and (2) effective fusion considering high-order relevance modeling. Concretely, we first propose a group multi-head self-attention gene encoder to capture global structured features in gene expression cohorts. We design a masked patch modeling paradigm (MPM) to capture the latent pathological characteristics of different tissues. The mask strategy is randomly masking a fixed-length contiguous subsequence of patch embeddings of a WSI. Finally, we combine the classification tokens of paired modalities and propose a triplet learning module to learn high-order relevance and discriminative patient-level information.After pre-training, a simple fine-tuning can be adopted to obtain the classification results. Experimental results on the TCGA dataset show the superiority of our network architectures and our pre-training framework, achieving 99.47% in accuracy for image-omic classification. The code is publicly available at https://github.com/huangwudiduan/GIMP.

翻译：肿瘤微环境组织学分析与基因组检测相结合是现代医学中大多数癌症的金标准。本文提出了一种基因诱导的多模态预训练（GiMP）框架，该框架将基因组学与全切片图像（WSIs）联合用于分类任务。我们的工作旨在解决多模态图像-组学分类的主要挑战，包括：（1）从千兆像素级别的WSIs和数万个基因中提取患者级别特征的困难；（2）考虑高阶相关性建模的有效融合。具体地，我们首先提出一种分组多头自注意力基因编码器，以捕获基因表达谱中的全局结构化特征。我们设计了一种掩码块建模范式（MPM），用于捕获不同组织的潜在病理特征。掩码策略是随机掩码WSI中固定长度的连续子序列块嵌入。最后，我们结合配对模态的分类令牌，并提出一种三元组学习模块，以学习高阶相关性和具有判别性的患者级别信息。预训练后，可通过简单微调获得分类结果。在TCGA数据集上的实验结果表明，我们的网络架构和预训练框架具有优越性，在图像-组学分类任务中达到了99.47%的准确率。代码已在https://github.com/huangwudiduan/GIMP 公开。