Uncertainty around predictions from a model due to the finite size of the development sample has traditionally been approached using classical inferential techniques. The finite size of the sample will result in the discrepancy between the final model and the correct model that maps covariates to predicted risks. From a decision-theoretic perspective, this discrepancy might affect the subsequent treatment decisions, and thus is associated with utility loss. From this perspective, procuring more development data is associated in expected gain in the utility of using the model. In this work, we define the Expected Value of Sample Information (EVSI) as the expected gain in clinical utility, defined in net benefit (NB) terms in net true positive units, by procuring a further development sample of a given size. We propose a bootstrap-based algorithm for EVSI computations, and show its feasibility and face validity in a case study. Decision-theoretic metrics can complement classical inferential methods when designing studies that are aimed at developing risk prediction models.

翻译：由于开发样本规模有限，模型预测存在不确定性，传统上采用经典推断技术处理这一问题。样本规模有限会导致最终模型与真实协变量-风险映射模型之间存在差异。从决策论视角看，这种差异可能影响后续治疗决策，从而造成效用损失。基于此视角，获取更多开发数据意味着模型使用效用的预期增益。本研究将样本信息期望值定义为：通过获取特定规模的额外开发样本，以净真阳性单位表示的净效益所衡量的临床效用预期增益。我们提出基于自助法的EVSI计算算法，并通过案例研究验证其可行性与表面效度。在设计风险预测模型开发研究时，决策论指标可作为经典推断方法的有效补充。