Federated learning aims to train models collaboratively across different clients without the sharing of data for privacy considerations. However, one major challenge for this learning paradigm is the {\em data heterogeneity} problem, which refers to the discrepancies between the local data distributions among various clients. To tackle this problem, we first study how data heterogeneity affects the representations of the globally aggregated models. Interestingly, we find that heterogeneous data results in the global model suffering from severe {\em dimensional collapse}, in which representations tend to reside in a lower-dimensional space instead of the ambient space. Moreover, we observe a similar phenomenon on models locally trained on each client and deduce that the dimensional collapse on the global model is inherited from local models. In addition, we theoretically analyze the gradient flow dynamics to shed light on how data heterogeneity result in dimensional collapse for local models. To remedy this problem caused by the data heterogeneity, we propose {\sc FedDecorr}, a novel method that can effectively mitigate dimensional collapse in federated learning. Specifically, {\sc FedDecorr} applies a regularization term during local training that encourages different dimensions of representations to be uncorrelated. {\sc FedDecorr}, which is implementation-friendly and computationally-efficient, yields consistent improvements over baselines on standard benchmark datasets. Code: https://github.com/bytedance/FedDecorr.

翻译：联邦学习旨在通过跨不同客户端协作训练模型，同时出于隐私考虑不共享数据。然而，该学习范式面临的一大挑战是**数据异质性**问题，即不同客户端之间本地数据分布的差异。为解决此问题，我们首先研究了数据异质性如何影响全局聚合模型的表征。有趣的是，我们发现异质性数据会导致全局模型遭受严重的**维度坍塌**，即表征倾向于位于低维空间而非环境空间中。此外，我们观察到每个客户端本地训练的模型也存在类似现象，并推断全局模型上的维度坍塌是继承自本地模型。同时，我们从理论上分析了梯度流动力学，以阐明数据异质性如何导致本地模型的维度坍塌。为缓解数据异质性引发的这一问题，我们提出**FedDecorr**，一种能有效减轻联邦学习中维度坍塌的新方法。具体而言，**FedDecorr**在本地训练过程中引入正则化项，促使表征的不同维度之间不相关。**FedDecorr**实现简单且计算高效，在标准基准数据集上持续优于基线方法。代码：https://github.com/bytedance/FedDecorr。