Medical applications of machine learning (ML) have experienced a surge in popularity in recent years. The intensive care unit (ICU) is a natural habitat for ML given the abundance of available data from electronic health records. Models have been proposed to address numerous ICU prediction tasks like the early detection of complications. While authors frequently report state-of-the-art performance, it is challenging to verify claims of superiority. Datasets and code are not always published, and cohort definitions, preprocessing pipelines, and training setups are difficult to reproduce. This work introduces Yet Another ICU Benchmark (YAIB), a modular framework that allows researchers to define reproducible and comparable clinical ML experiments; we offer an end-to-end solution from cohort definition to model evaluation. The framework natively supports most open-access ICU datasets (MIMIC III/IV, eICU, HiRID, AUMCdb) and is easily adaptable to future ICU datasets. Combined with a transparent preprocessing pipeline and extensible training code for multiple ML and deep learning models, YAIB enables unified model development. Our benchmark comes with five predefined established prediction tasks (mortality, acute kidney injury, sepsis, kidney function, and length of stay) developed in collaboration with clinicians. Adding further tasks is straightforward by design. Using YAIB, we demonstrate that the choice of dataset, cohort definition, and preprocessing have a major impact on the prediction performance - often more so than model class - indicating an urgent need for YAIB as a holistic benchmarking tool. We provide our work to the clinical ML community to accelerate method development and enable real-world clinical implementations. Software Repository: https://github.com/rvandewater/YAIB.

翻译：机器学习在医学领域的应用近年来急剧增长。重症监护病房（ICU）因其电子健康记录中丰富的可用数据，成为机器学习的天然应用场景。研究者已提出多种模型用于解决ICU预测任务，如并发症的早期检测。尽管作者们常报告其模型具有最先进的性能，但验证其优越性声明颇具挑战性。数据集和代码并非总是公开，且队列定义、预处理流程和训练设置难以复现。本研究提出“又一个ICU基准”（YAIB）——一个模块化框架，使研究者能够定义可复现且可比较的临床机器学习实验。我们提供从队列定义到模型评估的端到端解决方案。该框架原生支持大多数开放获取的ICU数据集（MIMIC III/IV、eICU、HiRID、AUMCdb），并可轻松适配未来的ICU数据集。结合透明的预处理流水线及针对多种机器学习和深度学习模型的可扩展训练代码，YAIB实现了统一的模型开发。我们的基准包含与临床医生合作开发的五个预定义经典预测任务（死亡率、急性肾损伤、脓毒症、肾功能及住院时长）。通过设计，添加新任务可直接实现。利用YAIB，我们证明数据集选择、队列定义和预处理对预测性能有重大影响——其影响常超过模型类别本身——这表明亟需YAIB作为整体性基准工具。我们将此项工作提供给临床机器学习社区，以加速方法开发并推动实际临床部署。软件仓库：https://github.com/rvandewater/YAIB。