Electronic health records (EHRs) recorded in hospital settings typically contain a wide range of numeric time series data that is characterized by high sparsity and irregular observations. Effective modelling for such data must exploit its time series nature, the semantic relationship between different types of observations, and information in the sparsity structure of the data. Self-supervised Transformers have shown outstanding performance in a variety of structured tasks in NLP and computer vision. But multivariate time series data contains structured relationships over two dimensions: time and recorded event type, and straightforward applications of Transformers to time series data do not leverage this distinct structure. The quadratic scaling of self-attention layers can also significantly limit the input sequence length without appropriate input engineering. We introduce the DuETT architecture, an extension of Transformers designed to attend over both time and event type dimensions, yielding robust representations from EHR data. DuETT uses an aggregated input where sparse time series are transformed into a regular sequence with fixed length; this lowers the computational complexity relative to previous EHR Transformer models and, more importantly, enables the use of larger and deeper neural networks. When trained with self-supervised prediction tasks, that provide rich and informative signals for model pre-training, our model outperforms state-of-the-art deep learning models on multiple downstream tasks from the MIMIC-IV and PhysioNet-2012 EHR datasets.

翻译：摘要：医院环境中的电子健康记录（EHR）通常包含大量数值型时间序列数据，其特征为高度稀疏性和非规则观测。对此类数据的有效建模需充分利用其时间序列特性、不同类型观测之间的语义关系以及数据稀疏结构中的信息。自监督变换器在自然语言处理和计算机视觉的多种结构化任务中展现了卓越性能。然而，多变量时间序列数据在时间和记录事件类型两个维度上存在结构化关系，而将变换器直接应用于时间序列数据无法利用这一独特结构。自注意力层的二次复杂度若不经过适当的输入工程处理，会显著限制输入序列长度。我们提出DuETT架构，这是对变换器的扩展，旨在同时关注时间和事件类型两个维度，从而从EHR数据中生成稳健的表示。DuETT采用聚合输入方式，将稀疏时间序列转换为固定长度的规则序列；这降低了与先前EHR变换器模型相比的计算复杂度，更重要的是，能够使用更大更深的神经网络。通过自监督预测任务（为模型预训练提供丰富且有信息量的信号）进行训练后，我们的模型在MIMIC-IV和PhysioNet-2012 EHR数据集的多个下游任务上超越了最先进的深度学习模型。