In most prediction and estimation situations, scientists consider various statistical models for the same problem, and naturally want to select amongst the best. Hansen et al. (2011) provide a powerful solution to this problem by the so-called model confidence set, a subset of the original set of available models that contains the best models with a given level of confidence. Importantly, model confidence sets respect the underlying selection uncertainty by being flexible in size. However, they presuppose a fixed sample size which stands in contrast to the fact that model selection and forecast evaluation are inherently sequential tasks where we successively collect new data and where the decision to continue or conclude a study may depend on the previous outcomes. In this article, we extend model confidence sets sequentially over time by relying on sequential testing methods. Recently, e-processes and confidence sequences have been introduced as new, safe methods for assessing statistical evidence. Sequential model confidence sets allow to continuously monitor the models' performances and come with time-uniform, nonasymptotic coverage guarantees.
翻译:在多数预测与估计场景中,研究者常针对同一问题考量多种统计模型,并自然期望从中筛选出最优模型。Hansen等人(2011)通过提出所谓模型置信集为此问题提供了强有力的解决方案——该集合作为原始可用模型集的子集,能以给定置信水平包含最优模型。值得注意的是,模型置信集通过规模可变性体现了潜在的选择不确定性。然而,该方法预设了固定样本量,这与模型选择及预测评估本质上属于序列任务的特性相悖:在这些任务中,我们持续收集新数据,且是否继续或终止研究的决策常取决于前期结果。本文借助序列检验方法,将模型置信集扩展为随时间推进的序列化框架。近期,e-过程与置信序列作为评估统计证据的新型安全方法被提出。序列模型置信集能够持续监测模型性能,并提供时间均匀、非渐近的覆盖保证。