Deep learning (DL) can aid doctors in detecting worsening patient states early, affording them time to react and prevent bad outcomes. While DL-based early warning models usually work well in the hospitals they were trained for, they tend to be less reliable when applied at new hospitals. This makes it difficult to deploy them at scale. Using carefully harmonised intensive care data from four data sources across Europe and the US (totalling 334,812 stays), we systematically assessed the reliability of DL models for three common adverse events: death, acute kidney injury (AKI), and sepsis. We tested whether using more than one data source and/or explicitly optimising for generalisability during training improves model performance at new hospitals. We found that models achieved high AUROC for mortality (0.838-0.869), AKI (0.823-0.866), and sepsis (0.749-0.824) at the training hospital. As expected, performance dropped at new hospitals, sometimes by as much as -0.200. Using more than one data source for training mitigated the performance drop, with multi-source models performing roughly on par with the best single-source model. This suggests that as data from more hospitals become available for training, model robustness is likely to increase, lower-bounding robustness with the performance of the most applicable data source in the training data. Dedicated methods promoting generalisability did not noticeably improve performance in our experiments.

翻译：深度学习（DL）可辅助医生及早识别患者病情恶化状态，为其争取干预时间并预防不良预后。尽管基于DL的早期预警模型通常在受训医院表现良好，但在新医疗机构应用时可靠性往往下降，这制约了其规模化部署。通过系统整合来自欧洲和美国四个数据源（共334,812次住院记录）的重症监护数据，我们系统评估了DL模型在三种常见不良事件（死亡、急性肾损伤（AKI）和脓毒症）中的可靠性。我们检验了以下假设：在训练过程中使用多源数据和/或显式优化模型普适性，是否可提升模型在新医院的表现。研究发现，模型在受训医院对死亡率（AUROC 0.838-0.869）、AKI（0.823-0.866）和脓毒症（0.749-0.824）均取得良好性能。正如预期，模型在新医院的性能出现下降，部分下降幅度达-0.200。采用多数据源训练可缓解性能衰减，多源模型的表现与最优单源模型大致相当。这表明随着更多医院数据可用于训练，模型鲁棒性将趋于提升，其性能下限受限于训练数据中最具适用性数据源的表现。在我们的实验中，专门提升普适性的方法并未带来显著性能改善。