External validation is often recommended to ensure the generalizability of ML models. However, it neither guarantees generalizability nor equates to a model's clinical usefulness (the ultimate goal of any clinical decision-support tool). External validation is misaligned with current healthcare ML needs. First, patient data changes across time, geography, and facilities. These changes create significant volatility in the performance of a single fixed model (especially for deep learning models, which dominate clinical ML). Second, newer ML techniques, current market forces, and updated regulatory frameworks are enabling frequent updating and monitoring of individual deployed model instances. We submit that external validation is insufficient to establish ML models' safety or utility. Proposals to fix the external validation paradigm do not go far enough. Continued reliance on it as the ultimate test is likely to lead us astray. We propose the MLOps-inspired paradigm of recurring local validation as an alternative that ensures the validity of models while protecting against performance-disruptive data variability. This paradigm relies on site-specific reliability tests before every deployment, followed by regular and recurrent checks throughout the life cycle of the deployed algorithm. Initial and recurrent reliability tests protect against performance-disruptive distribution shifts, and concept drifts that jeopardize patient safety.

翻译：外部验证常被推荐用于确保机器学习模型的泛化能力。然而，它既不能保证泛化性，也不等同于模型的临床实用性（任何临床决策支持工具的终极目标）。外部验证与当前医疗机器学习需求存在错位。首先，患者数据随时间、地域和机构而变化。这些变化会给单个固定模型的性能带来显著波动（尤其是主导临床ML的深度学习模型）。其次，较新的ML技术、当前市场力量以及更新的监管框架使得对单个已部署模型实例进行频繁更新和监测成为可能。我们认为外部验证不足以确立ML模型的安全性及效用。试图修补外部验证范式的提案尚不充分。继续将其作为最终检验标准很可能会导致误入歧途。我们提出基于MLOps理念的循环局部验证范式作为替代方案，该方案在保障模型有效性的同时，可防范破坏性能的数据变异。该范式要求在每次部署前进行针对特定站点的可靠性测试，并在算法全生命周期中开展定期循环检查。初始与循环可靠性测试可防御威胁患者安全的破坏性分布偏移与概念漂移。