Human readers or radiologists routinely perform full-body multi-organ multi-disease detection and diagnosis in clinical practice, while most medical AI systems are built to focus on single organs with a narrow list of a few diseases. This might severely limit AI's clinical adoption. A certain number of AI models need to be assembled non-trivially to match the diagnostic process of a human reading a CT scan. In this paper, we construct a Unified Tumor Transformer (CancerUniT) model to jointly detect tumor existence & location and diagnose tumor characteristics for eight major cancers in CT scans. CancerUniT is a query-based Mask Transformer model with the output of multi-tumor prediction. We decouple the object queries into organ queries, tumor detection queries and tumor diagnosis queries, and further establish hierarchical relationships among the three groups. This clinically-inspired architecture effectively assists inter- and intra-organ representation learning of tumors and facilitates the resolution of these complex, anatomically related multi-organ cancer image reading tasks. CancerUniT is trained end-to-end using a curated large-scale CT images of 10,042 patients including eight major types of cancers and occurring non-cancer tumors (all are pathology-confirmed with 3D tumor masks annotated by radiologists). On the test set of 631 patients, CancerUniT has demonstrated strong performance under a set of clinically relevant evaluation metrics, substantially outperforming both multi-disease methods and an assembly of eight single-organ expert models in tumor detection, segmentation, and diagnosis. This moves one step closer towards a universal high performance cancer screening tool.

翻译：在临床实践中，影像科医生或放射科医师通常需要对全身多器官、多疾病进行检测与诊断，而绝大多数医学人工智能系统仅针对单一器官及少量疾病设计，这严重限制了AI的临床应用潜力——若要匹配人类阅读CT扫描的诊断过程，需非平凡地集成多个AI模型。本文构建了统一肿瘤Transformer模型（CancerUniT），用于联合检测CT扫描中八种主要癌症的肿瘤存在性及位置，并诊断肿瘤特征。CancerUniT是基于查询的掩码Transformer模型，可输出多肿瘤预测结果。我们将目标查询解耦为器官查询、肿瘤检测查询和肿瘤诊断查询，并建立三组查询间的层级关系。这种受临床启发的架构有效促进了肿瘤的跨器官与器官内表征学习，解决了这些解剖学关联的复杂多器官癌症影像判读任务。CancerUniT采用包含10,042例患者的精选大规模CT影像数据集进行端到端训练（涵盖八种主要癌症及非癌性肿瘤，所有病例均经病理证实并附带放射科医生标注的三维肿瘤掩膜）。在包含631例患者的测试集上，CancerUniT在一系列临床相关评估指标中展现出强劲性能，在肿瘤检测、分割与诊断任务上显著优于多疾病联合方法及八种单器官专家模型的集成系统。这标志着我们向实现通用高性能癌症筛查工具迈出了重要一步。