Generalization performance is a key metric in evaluating machine learning models when applied to real-world applications. Good generalization indicates the model can predict unseen data correctly when trained under a limited number of data. Federated learning (FL), which has emerged as a popular distributed learning framework, allows multiple devices or clients to train a shared model without violating privacy requirements. While the existing literature has studied extensively the generalization performances of centralized machine learning algorithms, similar analysis in the federated settings is either absent or with very restrictive assumptions on the loss functions. In this paper, we aim to analyze the generalization performances of federated learning by means of algorithmic stability, which measures the change of the output model of an algorithm when perturbing one data point. Three widely-used algorithms are studied, including FedAvg, SCAFFOLD, and FedProx, under convex and non-convex loss functions. Our analysis shows that the generalization performances of models trained by these three algorithms are closely related to the heterogeneity of clients' datasets as well as the convergence behaviors of the algorithms. Particularly, in the i.i.d. setting, our results recover the classical results of stochastic gradient descent (SGD).

翻译：泛化性能是评估机器学习模型在现实世界应用中表现的关键指标。良好的泛化能力意味着模型在有限数据训练下能够正确预测未见数据。联邦学习作为新兴的分布式学习框架，允许多个设备或客户端在满足隐私要求的前提下训练共享模型。现有文献广泛研究了集中式机器学习算法的泛化性能，但在联邦设置下的类似分析要么缺失，要么对损失函数施加了严格限制。本文旨在通过算法稳定性（衡量算法在扰动一个数据点时输出模型的变化）来分析联邦学习的泛化性能。我们研究了三种广泛使用的算法——FedAvg、SCAFFOLD和FedProx——在凸和非凸损失函数下的表现。分析表明，这三种算法训练的模型的泛化性能与客户端数据集的异质性以及算法的收敛行为密切相关。特别地，在独立同分布设置下，我们的结果恢复了随机梯度下降（SGD）的经典结论。