Processing giga-pixel whole slide histopathology images (WSI) is a computationally expensive task. Multiple instance learning (MIL) has become the conventional approach to process WSIs, in which these images are split into smaller patches for further processing. However, MIL-based techniques ignore explicit information about the individual cells within a patch. In this paper, by defining the novel concept of shared-context processing, we designed a multi-modal Graph Transformer (AMIGO) that uses the celluar graph within the tissue to provide a single representation for a patient while taking advantage of the hierarchical structure of the tissue, enabling a dynamic focus between cell-level and tissue-level information. We benchmarked the performance of our model against multiple state-of-the-art methods in survival prediction and showed that ours can significantly outperform all of them including hierarchical Vision Transformer (ViT). More importantly, we show that our model is strongly robust to missing information to an extent that it can achieve the same performance with as low as 20% of the data. Finally, in two different cancer datasets, we demonstrated that our model was able to stratify the patients into low-risk and high-risk groups while other state-of-the-art methods failed to achieve this goal. We also publish a large dataset of immunohistochemistry images (InUIT) containing 1,600 tissue microarray (TMA) cores from 188 patients along with their survival information, making it one of the largest publicly available datasets in this context.

翻译：处理十亿像素级全切片组织病理学图像（WSI）是一项计算成本高昂的任务。多实例学习（MIL）已成为处理WSI的常规方法，这类方法将图像分割成更小的图块以进行后续处理。然而，基于MIL的技术忽略了图块内单个细胞的显式信息。本文通过定义共享上下文处理这一新概念，设计了一种多模态图Transformer（AMIGO），它利用组织内的细胞图在保留组织层次结构的同时为患者提供单一表征，从而实现对细胞级与组织级信息的动态聚焦。我们在生存预测任务中将模型性能与多种最先进方法进行基准测试，结果表明我们的模型显著优于所有对比方法，包括层级式Vision Transformer（ViT）。更重要的是，我们证明了模型对信息缺失具有极强的鲁棒性，即便仅使用20%的数据也能达到同等性能。最后，在两个不同的癌症数据集中，我们展示出模型能够将患者分层为低风险组与高风险组，而其他最先进方法均未能实现这一目标。我们还发布了一个包含188名患者1,600个组织微阵列（TMA）核心及其生存信息的大型免疫组化图像数据集（InUIT），使其成为该领域最大的公开数据集之一。