We investigate model assessment and selection in a changing environment, by synthesizing datasets from both the current time period and historical epochs. To tackle unknown and potentially arbitrary temporal distribution shift, we develop an adaptive rolling window approach to estimate the generalization error of a given model. This strategy also facilitates the comparison between any two candidate models by estimating the difference of their generalization errors. We further integrate pairwise comparisons into a single-elimination tournament, achieving near-optimal model selection from a collection of candidates. Theoretical analyses and numerical experiments demonstrate the adaptivity of our proposed methods to the non-stationarity in data.

翻译：我们通过整合当前时间段与历史时期的数据集，研究动态环境下的模型评估与选择问题。为应对未知且可能任意的时态分布偏移，我们开发了一种自适应滚动窗口方法，用于估计给定模型的泛化误差。该策略还可通过估计任意两个候选模型泛化误差的差异来促进模型间的比较。我们进一步将成对比较整合到单败淘汰锦标赛中，实现了从候选集合中近乎最优的模型选择。理论分析与数值实验表明，所提方法对数据非平稳性具有自适应性。