Prediction models provide risks of an adverse event occurring for an individual based on their characteristics. Some prediction models have been used to make treatment decisions, but this is not appropriate when the data on which the model was developed included a mix of individuals with some who did and others who did not initiate that treatment. By contrast, predictions under hypothetical interventions are estimates of what a person's risk of an outcome would be if they were to follow a particular treatment strategy, given their individual characteristics. Such predictions can give important input to medical decision making. However, evaluating predictive performance of interventional predictions is challenging. Standard ways of evaluating predictive performance do not apply, because prediction under interventions involves obtaining predictions of the outcome under conditions that differ from those that are observed for some patients in the validation data. This work describes methods for evaluating predictive performance of predictions under interventions using longitudinal observational data. We focus on time-to-event outcomes and predictions under treatment strategies that involve sustaining a particular treatment regime over time. We introduce a validation approach using artificial censoring and inverse probability weighting which involves creating a validation data set that mimics the particular treatment strategy under which predictions are made. We extend measures of calibration, discrimination and overall prediction error to the interventional prediction setting. The methods are evaluated using a simulation study and results show that our proposed approach and corresponding measures of predictive performance correctly capture the true predictive performance. The methods are applied to an example in the context of liver transplantation.

翻译：预测模型根据个体特征提供其发生不良事件的风险。部分预测模型已被用于制定治疗决策，但当模型开发所用数据包含既接受治疗又未接受治疗的混合人群时，这种应用并不恰当。相比之下，假设干预下的预测是一种估计：在给定个体特征的前提下，如果该个体遵循特定治疗策略，其发生结局事件的风险会是多少。此类预测可为医疗决策提供重要依据。然而，评估干预性预测的预测性能具有挑战性。由于干预预测需在不同于验证数据中部分患者实际观察到的条件下获取结局预测值，标准的预测性能评估方法不再适用。本文描述了利用纵向观察数据评估干预条件下预测性能的方法。我们重点关注时间至事件结局以及需长期维持特定治疗方案的策略下的预测。本文提出一种结合人工删失与逆概率加权的验证方法——通过构建模拟目标治疗策略的验证数据集，使预测条件与实际分析条件一致。我们将校准度、区分度及总体预测误差等指标拓展至干预预测场景。通过模拟研究验证，结果表明所提方法及其对应的预测性能指标能准确捕捉真实预测性能。最后将该方法应用于肝移植领域的实例分析。