Federated learning (FL) is a new paradigm for distributed machine learning that allows a global model to be trained across multiple clients without compromising their privacy. Although FL has demonstrated remarkable success in various scenarios, recent studies mainly utilize shallow and small neural networks. In our research, we discover a significant performance decline when applying the existing FL framework to deeper neural networks, even when client data are independently and identically distributed (i.i.d.). Our further investigation shows that the decline is due to the continuous accumulation of dissimilarities among client models during the layer-by-layer back-propagation process, which we refer to as "divergence accumulation." As deeper models involve a longer chain of divergence accumulation, they tend to manifest greater divergence, subsequently leading to performance decline. Both theoretical derivations and empirical evidence are proposed to support the existence of divergence accumulation and its amplified effects in deeper models. To address this issue, we propose several technical guidelines based on reducing divergence, such as using wider models and reducing the receptive field. These approaches can greatly improve the accuracy of FL on deeper models. For example, the application of these guidelines can boost the ResNet101 model's performance by as much as 43\% on the Tiny-ImageNet dataset.

翻译：联邦学习（FL）是一种分布式机器学习的新范式，允许在多个客户端上训练全局模型，同时保护其隐私。尽管FL已在多种场景中展现出显著成功，但近期研究主要利用浅层和小型神经网络。在我们的研究中，发现将现有FL框架应用于深层神经网络时会出现显著性能下降，即使客户端数据是独立同分布的（i.i.d.）。进一步研究表明，这种下降源于逐层反向传播过程中客户端模型间差异的持续累积，我们称之为“差异积累”。由于深层模型涉及更长的差异积累链，它们往往表现出更大的差异，进而导致性能下降。我们提出了理论推导和实验证据来支持差异积累的存在及其在深层模型中的放大效应。为解决该问题，我们提出了基于减少差异的若干技术指南，例如使用更宽的模型和缩小感受野。这些方法能显著提升FL在深层模型上的准确性。例如，应用这些指南可使Tiny-ImageNet数据集上的ResNet101模型性能提升高达43%。