External validation is often recommended to ensure the generalizability of ML models. However, it neither guarantees generalizability nor equates to a model's clinical usefulness (the ultimate goal of any clinical decision-support tool). External validation is misaligned with current healthcare ML needs. First, patient data changes across time, geography, and facilities. These changes create significant volatility in the performance of a single fixed model (especially for deep learning models, which dominate clinical ML). Second, newer ML techniques, current market forces, and updated regulatory frameworks are enabling frequent updating and monitoring of individual deployed model instances. We submit that external validation is insufficient to establish ML models' safety or utility. Proposals to fix the external validation paradigm do not go far enough. Continued reliance on it as the ultimate test is likely to lead us astray. We propose the MLOps-inspired paradigm of recurring local validation as an alternative that ensures the validity of models while protecting against performance-disruptive data variability. This paradigm relies on site-specific reliability tests before every deployment, followed by regular and recurrent checks throughout the life cycle of the deployed algorithm. Initial and recurrent reliability tests protect against performance-disruptive distribution shifts, and concept drifts that jeopardize patient safety.

翻译：外部验证常被推荐用于确保机器学习模型的泛化能力，但它既无法保证泛化性，也不等同于模型的临床实用性（任何临床决策支持工具的终极目标）。外部验证与当前医疗健康领域机器学习的需求存在错位。首先，患者数据会因时间、地域和医疗机构的不同而变化，这些变化导致单一固定模型（尤其是主导临床机器学习的深度学习模型）的性能出现显著波动。其次，新兴的机器学习技术、当前市场力量及更新后的监管框架，正在推动个体部署模型实例的频繁更新和监控。我们认为，外部验证不足以建立机器学习模型的安全性或有效性。试图修补外部验证范式的方案远未触及根本。若继续将其作为终极检验标准，很可能让我们误入歧途。我们提出基于MLOps理念的循环局部验证范式作为替代方案，该范式既能确保模型有效性，又能防范性能破坏性的数据变异。该范式要求在每次部署前进行特定站点的可靠性测试，随后在算法全生命周期中定期循环检测。初始及循环可靠性测试可抵御危及患者安全的性能破坏性分布偏移和概念漂移。