Despite the proven effectiveness of Transformer neural networks across multiple domains, their performance with Electronic Health Records (EHR) can be nuanced. The unique, multidimensional sequential nature of EHR data can sometimes make even simple linear models with carefully engineered features more competitive. Thus, the advantages of Transformers, such as efficient transfer learning and improved scalability are not always fully exploited in EHR applications. Addressing these challenges, we introduce SANSformer, an attention-free sequential model designed with specific inductive biases to cater for the unique characteristics of EHR data. In this work, we aim to forecast the demand for healthcare services, by predicting the number of patient visits to healthcare facilities. The challenge amplifies when dealing with divergent patient subgroups, like those with rare diseases, which are characterized by unique health trajectories and are typically smaller in size. To address this, we employ a self-supervised pretraining strategy, Generative Summary Pretraining (GSP), which predicts future summary statistics based on past health records of a patient. Our models are pretrained on a health registry of nearly one million patients, then fine-tuned for specific subgroup prediction tasks, showcasing the potential to handle the multifaceted nature of EHR data. In evaluation, SANSformer consistently surpasses robust EHR baselines, with our GSP pretraining method notably amplifying model performance, particularly within smaller patient subgroups. Our results illuminate the promising potential of tailored attention-free models and self-supervised pretraining in refining healthcare utilization predictions across various patient demographics.

翻译：尽管Transformer神经网络在多个领域已被证明有效，但其在电子健康记录（EHR）中的应用效果可能较为复杂。EHR数据独特的多维序列特性，有时甚至会使精心设计特征工程的简单线性模型更具竞争力。因此，Transformer的优势（如高效的迁移学习和改进的可扩展性）在EHR应用中并未得到充分利用。针对这些挑战，我们提出SANSformer——一种专为适配EHR数据特性而设计、包含特定归纳偏置的无注意力序列模型。本研究旨在通过预测患者就诊医疗机构的数量，实现医疗服务的需求预测。当面对差异化的患者亚组（如以独特健康轨迹和较小规模为特征的罕见病患者）时，这一挑战会进一步加剧。为此，我们采用自监督预训练策略——生成式摘要预训练（GSP），通过基于患者既往健康记录预测未来摘要统计量。我们的模型在包含近百万名患者的健康登记数据库上进行预训练，随后针对特定亚组预测任务进行微调，充分展示了其处理EHR数据多层面特性的潜力。评估结果表明，SANSformer持续超越稳健的EHR基线模型，其中GSP预训练方法显著提升了模型性能，尤其在小规模患者亚组中表现突出。研究结果揭示了定制化无注意力模型与自监督预训练在优化跨患者群体医疗服务利用率预测方面的广阔前景。