Cytopathology report generation is a necessary step for the standardized examination of pathology images. However, manually writing detailed reports brings heavy workloads for pathologists. To improve efficiency, some existing works have studied automatic generation of cytopathology reports, mainly by applying image caption generation frameworks with visual encoders originally proposed for natural images. A common weakness of these works is that they do not explicitly model the structural information among cells, which is a key feature of pathology images and provides significant information for making diagnoses. In this paper, we propose a novel graph-based framework called GNNFormer, which seamlessly integrates graph neural network (GNN) and Transformer into the same framework, for cytopathology report generation. To the best of our knowledge, GNNFormer is the first report generation method that explicitly models the structural information among cells in pathology images. It also effectively fuses structural information among cells, fine-grained morphology features of cells and background features to generate high-quality reports. Experimental results on the NMI-WSI dataset show that GNNFormer can outperform other state-of-the-art baselines.

翻译：细胞病理学报告生成是病理图像标准化检查的必要步骤。然而，手动撰写详细报告给病理学家带来了沉重的工作负担。为提高效率，已有部分研究探索了自动生成细胞病理学报告的方法，主要通过应用最初为自然图像设计的视觉编码器所实现的图像描述生成框架。这些研究的一个共同缺陷是未显式建模细胞间的结构信息——这是病理图像的关键特征，为诊断提供了重要依据。本文提出了一种新颖的基于图的框架GNNFormer，它将图神经网络（GNN）与Transformer无缝集成于同一框架中，用于细胞病理学报告生成。据我们所知，GNNFormer是首个显式建模病理图像中细胞间结构信息的报告生成方法。该方法有效融合了细胞间的结构信息、细胞的细粒度形态特征以及背景特征，以生成高质量报告。在NMI-WSI数据集上的实验结果表明，GNNFormer能够优于其他最先进的基线方法。