Many pre-trained models (PTMs) are available in modern applications. Because different PTMs are often trained on different datasets, their performances can vary substantially for different new tasks, and the ranking of the candidates may depend heavily on the input. Motivated by this, we propose a localized model averaging method with weights modeled as functions of the covariates, making it substantially more versatile than existing model averaging methods. This formulation allows the model averaging procedure to adaptively capture the varying relative advantages of different PTMs across heterogeneous contexts. Specifically, we learn flexible local weights under a general loss framework that accommodates a broad class of prediction tasks. We further establish the asymptotic optimality of the proposed method for both in-sample and out-of-sample risks, as well as the consistency of the estimated weights. Extensive numerical experiments further demonstrate the effectiveness of the proposed method.

翻译：现代应用中存在大量预训练模型。由于不同预训练模型通常在不同数据集上训练，它们在不同新任务上的表现可能差异显著，且候选模型的排名高度依赖于输入数据。受此启发，我们提出一种局部模型平均方法，其权重被建模为协变量的函数，这使得该方法比现有模型平均方法更具通用性。该公式允许模型平均过程自适应地捕捉不同预训练模型在异质情境下的相对优势变化。具体而言，我们在适用于广泛预测任务的通用损失框架下学习灵活的局部权重。我们进一步证明了所提方法在样本内和样本外风险方面的渐近最优性，以及估计权重的一致性。大量数值实验也验证了所提方法的有效性。