The application of Transformer neural networks to Electronic Health Records (EHR) is challenging due to the distinct, multidimensional sequential structure of EHR data, often leading to underperformance when compared to simpler linear models. Thus, the advantages of Transformers, such as efficient transfer learning and improved scalability are not fully exploited in EHR applications. To overcome these challenges, we introduce SANSformer, a novel attention-free sequential model designed specifically with inductive biases to cater for the unique characteristics of EHR data. Our main application area is predicting future healthcare utilization, a crucial task for effectively allocating healthcare resources. This task becomes particularly difficult when dealing with divergent patient subgroups. These subgroups, characterized by unique health trajectories and often small in size, such as patients with rare diseases, require specialized modeling approaches. To address this, we adopt a self-supervised pretraining strategy, which we term Generative Summary Pretraining (GSP). GSP predicts summary statistics of a future window in the patient's history based on their past health records, thus demonstrating potential to deal with the noisy and complex nature of EHR data. We pretrain our models on a comprehensive health registry encompassing close to one million patients, before fine-tuning them for specific subgroup prediction tasks. In our evaluations, SANSformer consistently outshines strong EHR baselines. Importantly, our GSP pretraining method greatly enhances model performance, especially for smaller patient subgroups. Our findings underscore the substantial potential of bespoke attention-free models and self-supervised pretraining for enhancing healthcare utilization predictions across a broad range of patient groups.

翻译：摘要：将Transformer神经网络应用于电子健康记录（EHR）面临挑战，这是由于EHR数据具有独特的多维序列结构，往往导致其性能不如简单的线性模型。因此，Transformer的优势（如高效迁移学习和改进的可扩展性）在EHR应用中未能得到充分发挥。为克服这些难题，我们提出SANSformer——一种专门针对EHR数据特征设计归纳偏置的新型无注意力机制序列模型。其主要应用领域是预测未来医疗资源利用率，这对有效配置医疗资源至关重要。当处理具有不同健康轨迹且样本量较小的患者亚群（如罕见病患者）时，该任务尤为困难。为此，我们采用自监督预训练策略，即生成式摘要预训练（GSP）。GSP通过基于患者既往病史预测未来时间窗口的汇总统计量，展现了应对EHR数据噪声与复杂性的潜力。我们在涵盖近百万患者的综合健康登记数据集上预训练模型，随后针对特定亚群预测任务进行微调。评估结果表明，SANSformer始终优于强大的EHR基线模型。重要的是，GSP预训练方法显著提升了模型性能，尤其对规模较小的患者亚群效果更佳。我们的发现凸显了定制化无注意力机制模型与自监督预训练在提升广泛患者群体医疗资源利用率预测方面的巨大潜力。