Federated learning (FL) inevitably confronts the challenge of system heterogeneity in practical scenarios. To enhance the capabilities of most model-homogeneous FL methods in handling system heterogeneity, we propose a training scheme that can extend their capabilities to cope with this challenge. In this paper, we commence our study with a detailed exploration of homogeneous and heterogeneous FL settings and discover three key observations: (1) a positive correlation between client performance and layer similarities, (2) higher similarities in the shallow layers in contrast to the deep layers, and (3) the smoother gradients distributions indicate the higher layer similarities. Building upon these observations, we propose InCo Aggregation that leverages internal cross-layer gradients, a mixture of gradients from shallow and deep layers within a server model, to augment the similarity in the deep layers without requiring additional communication between clients. Furthermore, our methods can be tailored to accommodate model-homogeneous FL methods such as FedAvg, FedProx, FedNova, Scaffold, and MOON, to expand their capabilities to handle the system heterogeneity. Copious experimental results validate the effectiveness of InCo Aggregation, spotlighting internal cross-layer gradients as a promising avenue to enhance the performance in heterogeneous FL.

翻译：联邦学习（FL）在实际场景中不可避免面临系统异质性的挑战。为增强大多数模型同质性FL方法处理系统异质性的能力，我们提出一种训练方案，可将其能力扩展至应对这一挑战。本文通过详细探究同质性与异质性联邦学习设置，发现三个关键现象：（1）客户端性能与层相似性呈正相关，（2）浅层相比深层具有更高的相似性，（3）更平滑的梯度分布对应更高的层相似性。基于这些发现，我们提出InCo聚合方法，利用服务器模型内部浅层与深层的梯度混合（即内部跨层梯度）来增强深层相似性，且无需客户端间额外通信。进一步地，我们的方法可针对性地适配FedAvg、FedProx、FedNova、Scaffold和MOON等模型同质性FL方法，扩展其处理系统异质性的能力。大量实验结果验证了InCo聚合的有效性，凸显内部跨层梯度作为提升异质性FL性能的一条有前景的途径。