In healthcare, predictive models increasingly inform patient-level decisions, yet little attention is paid to the variability in individual risk estimates and its impact on treatment decisions. For overparameterized models, now standard in machine learning, a substantial source of variability often goes undetected. Even when the data and model architecture are held fixed, randomness introduced by optimization and initialization can lead to materially different risk estimates for the same patient. This problem is largely obscured by standard evaluation practices, which rely on aggregate performance metrics (e.g., log-loss, accuracy) that are agnostic to individual-level stability. As a result, models with indistinguishable aggregate performance can nonetheless exhibit substantial procedural arbitrariness, which can undermine clinical trust. We propose an evaluation framework that quantifies individual-level prediction instability by using two complementary diagnostics: empirical prediction interval width (ePIW), which captures variability in continuous risk estimates, and empirical decision flip rate (eDFR), which measures instability in threshold-based clinical decisions. We apply these diagnostics to simulated data and GUSTO-I clinical dataset. Across observed settings, we find that for flexible machine-learning models, randomness arising solely from optimization and initialization can induce individual-level variability comparable to that produced by resampling the entire training dataset. Neural networks exhibit substantially greater instability in individual risk predictions compared to logistic regression models. Risk estimate instability near clinically relevant decision thresholds can alter treatment recommendations. These findings that stability diagnostics should be incorporated into routine model validation for assessing clinical reliability.

翻译：在医疗领域，预测模型越来越多地用于指导患者层面的决策，但个体风险估计的变异性及其对治疗决策的影响却鲜受关注。对于当前机器学习中普遍采用的高度参数化模型而言，一种重要的变异性来源常被忽视：即便数据和模型架构保持不变，由优化过程和初始化引入的随机性也可能导致同一患者获得截然不同的风险估计。这一问题的隐蔽性源于标准评估实践——其依赖于对个体级稳定性不敏感的聚合性能指标（如对数损失、准确率）。因此，具有难以区分的聚合性能的模型仍可能表现出显著的程序性随意性，进而削弱临床信任。我们提出一种评估框架，通过两个互补诊断指标量化个体级预测不稳定性：经验预测区间宽度（ePIW）捕捉连续风险估计的变异性，经验决策翻转率（eDFR）衡量基于阈值的临床决策的不稳定性。我们将这些诊断方法应用于模拟数据和GUSTO-I临床数据集。在观测到的各类设置中，我们发现对于灵活型机器学习模型，仅由优化和初始化产生的随机性所诱导的个体级变异性，可与重采样整个训练数据集产生的变异性相当。相比逻辑回归模型，神经网络在个体风险预测中表现出显著更大的不稳定性。临床相关决策阈值附近的风险估计不稳定性可能改变治疗建议。这些发现表明，应将稳定性诊断纳入常规模型验证流程，以评估临床可靠性。