A few recent studies have demonstrated that leveraging centrally pre-trained models can offer advantageous initializations for federated learning (FL). However, existing pre-training methods do not generalize well when faced with an arbitrary set of downstream FL tasks. Specifically, they often (i) achieve limited average accuracy, particularly when there are unseen downstream labels, and (ii) result in significant accuracy variance, failing to provide a balanced performance across clients. To address these challenges, we propose CoPreFL, a collaborative/distributed pre-training approach which provides a robust initialization for downstream FL tasks. The key idea of CoPreFL is a model-agnostic meta-learning (MAML) procedure that tailors the global model to closely mimic heterogeneous and unseen FL scenarios, resulting in a pre-trained model that is rapidly adaptable to arbitrary FL tasks. Our MAML procedure incorporates performance variance into the meta-objective function, balancing performance across clients rather than solely optimizing for accuracy. Through extensive experiments, we demonstrate that CoPreFL obtains significant improvements in both average accuracy and variance across arbitrary downstream FL tasks with unseen/seen labels, compared with various pre-training baselines. We also show how CoPreFL is compatible with different well-known FL algorithms applied by the downstream tasks, enhancing performance in each case.

翻译：近期少数研究表明，利用中心化预训练模型可为联邦学习（FL）提供有利的初始化条件。然而，现有预训练方法在面对任意下游FL任务集时泛化能力不足。具体表现为：（i）平均准确率有限，尤其当下游任务包含未见标签时；（ii）准确率方差显著，无法在客户端间提供均衡性能。为应对这些挑战，我们提出CoPreFL——一种为下游FL任务提供鲁棒初始化的协同/分布式预训练方法。CoPreFL的核心思想是通过模型无关元学习（MAML）流程，使全局模型能精准模拟异构且未知的FL场景，从而获得可快速适应任意FL任务的预训练模型。我们的MAML流程将性能方差纳入元目标函数，在客户端间实现性能均衡而非单纯优化准确率。大量实验表明，与多种预训练基线方法相比，CoPreFL在包含未见/已知标签的任意下游FL任务中，均能显著提升平均准确率并降低方差。我们还验证了CoPreFL可与下游任务采用的不同经典FL算法兼容，并在各种场景中提升其性能。