The accuracy of predictive models for solitary pulmonary nodule (SPN) diagnosis can be greatly increased by incorporating repeat imaging and medical context, such as electronic health records (EHRs). However, clinically routine modalities such as imaging and diagnostic codes can be asynchronous and irregularly sampled over different time scales which are obstacles to longitudinal multimodal learning. In this work, we propose a transformer-based multimodal strategy to integrate repeat imaging with longitudinal clinical signatures from routinely collected EHRs for SPN classification. We perform unsupervised disentanglement of latent clinical signatures and leverage time-distance scaled self-attention to jointly learn from clinical signatures expressions and chest computed tomography (CT) scans. Our classifier is pretrained on 2,668 scans from a public dataset and 1,149 subjects with longitudinal chest CTs, billing codes, medications, and laboratory tests from EHRs of our home institution. Evaluation on 227 subjects with challenging SPNs revealed a significant AUC improvement over a longitudinal multimodal baseline (0.824 vs 0.752 AUC), as well as improvements over a single cross-section multimodal scenario (0.809 AUC) and a longitudinal imaging-only scenario (0.741 AUC). This work demonstrates significant advantages with a novel approach for co-learning longitudinal imaging and non-imaging phenotypes with transformers. Code available at https://github.com/MASILab/lmsignatures.

翻译：孤立性肺结节（SPN）诊断的预测模型可通过整合重复影像及电子健康记录（EHRs）等医学背景信息显著提升准确性。然而，影像与诊断编码等临床常规模态在不同时间尺度上存在异步性和不规则采样问题，成为纵向多模态学习的障碍。本文提出基于Transformer的多模态策略，将重复影像与常规采集EHRs中的纵向临床特征相融合用于SPN分类。我们执行潜在临床特征的无监督解耦，并利用时间距离缩放自注意力机制联合学习临床特征表达与胸部计算机断层扫描（CT）影像。分类器在公开数据集的2668次扫描及本机构EHRs中1149名受试者的纵向胸部CT、账单代码、用药记录和实验室检查数据上完成预训练。针对227名具有挑战性SPN受试者的评估表明，相较于纵向多模态基线（AUC 0.824 vs 0.752），本方法在AUC上有显著提升，同时优于单截面多模态场景（AUC 0.809）和纵向纯影像场景（AUC 0.741）。本研究通过Transformer实现影像与非影像表型的协同学习，展示了新方法的显著优势。代码详见https://github.com/MASILab/lmsignatures。