Conventional federated learning algorithms train a single global model by leveraging all participating clients' data. However, due to heterogeneity in client generative distributions and predictive models, these approaches may not appropriately approximate the predictive process, converge to an optimal state, or generalize to new clients. We study personalization and generalization in stateless cross-device federated learning setups assuming heterogeneity in client data distributions and predictive models. We first propose a hierarchical generative model and formalize it using Bayesian Inference. We then approximate this process using Variational Inference to train our model efficiently. We call this algorithm Federated Variational Inference (FedVI). We use PAC-Bayes analysis to provide generalization bounds for FedVI. We evaluate our model on FEMNIST and CIFAR-100 image classification and show that FedVI beats the state-of-the-art on both tasks.

翻译：传统联邦学习算法通过利用所有参与客户端的数据来训练单个全局模型。然而，由于客户端生成分布和预测模型的异质性，这些方法可能无法准确逼近预测过程、收敛至最优状态，或泛化至新客户端。我们研究无状态跨设备联邦学习框架中数据分布与预测模型异质性下的个性化与泛化问题。首先提出一个层次生成模型，并利用贝叶斯推断对其进行形式化建模。随后采用变分推断近似该过程以高效训练模型，我们将此算法称为联邦变分推断（Federated Variational Inference, FedVI）。借助PAC-Bayes分析，我们推导出FedVI的泛化边界。在FEMNIST和CIFAR-100图像分类任务上的评估表明，FedVI在两项任务上均超越了现有最优方法。