The accuracy of predictive models for solitary pulmonary nodule (SPN) diagnosis can be greatly increased by incorporating repeat imaging and medical context, such as electronic health records (EHRs). However, clinically routine modalities such as imaging and diagnostic codes can be asynchronous and irregularly sampled over different time scales which are obstacles to longitudinal multimodal learning. In this work, we propose a transformer-based multimodal strategy to integrate repeat imaging with longitudinal clinical signatures from routinely collected EHRs for SPN classification. We perform unsupervised disentanglement of latent clinical signatures and leverage time-distance scaled self-attention to jointly learn from clinical signatures expressions and chest computed tomography (CT) scans. Our classifier is pretrained on 2,668 scans from a public dataset and 1,149 subjects with longitudinal chest CTs, billing codes, medications, and laboratory tests from EHRs of our home institution. Evaluation on 227 subjects with challenging SPNs revealed a significant AUC improvement over a longitudinal multimodal baseline (0.824 vs 0.752 AUC), as well as improvements over a single cross-section multimodal scenario (0.809 AUC) and a longitudinal imaging-only scenario (0.741 AUC). This work demonstrates significant advantages with a novel approach for co-learning longitudinal imaging and non-imaging phenotypes with transformers.

翻译：摘要：通过纳入重复影像检查和医疗背景信息（如电子健康记录），可显著提高孤立性肺结节（SPN）诊断预测模型的准确性。然而，临床上常规的影像学和诊断编码等模态往往存在时间异步性和不规则采样问题，这成为纵向多模态学习的障碍。本研究提出一种基于Transformer的多模态策略，将重复影像检查与从常规收集的电子病历中提取的纵向临床特征进行整合，用于SPN分类。我们通过无监督方式解耦潜在临床特征，并利用时间距离加权的自注意力机制，联合学习临床特征表达与胸部计算机断层扫描（CT）影像。所提出的分类器在公共数据集的2668次扫描和本机构电子病历中1149例具有纵向胸部CT、账单编码、药物及实验室检查结果的受试者上进行了预训练。针对227例具有挑战性SPN的受试者评估显示，相较于纵向多模态基线（AUC 0.824 vs 0.752）以及单一横断面多模态场景（AUC 0.809）和纵向仅影像场景（AUC 0.741），本研究方法在AUC上取得显著提升。本工作展示了利用Transformer协同学习纵向影像与非影像表型这一新方法的显著优势。