Federated learning is a promising framework to train neural networks with widely distributed data. However, performance degrades heavily with heterogeneously distributed data. Recent work has shown this is due to the final layer of the network being most prone to local bias, some finding success freezing the final layer as an orthogonal classifier. We investigate the training dynamics of the classifier by applying SVD to the weights motivated by the observation that freezing weights results in constant singular values. We find that there are differences when training in IID and non-IID settings. Based on this finding, we introduce two regularization terms for local training to continuously emulate IID settings: (1) variance in the dimension-wise probability distribution of the classifier and (2) hyperspherical uniformity of representations of the encoder. These regularizations promote local models to act as if it were in an IID setting regardless of the local data distribution, thus offsetting proneness to bias while being flexible to the data. On extensive experiments in both label-shift and feature-shift settings, we verify that our method achieves highest performance by a large margin especially in highly non-IID cases in addition to being scalable to larger models and datasets.

翻译：联邦学习是一种利用分布式数据训练神经网络的极具前景的框架。然而，当数据呈异构分布时，其性能会严重下降。近期研究表明，这是由于网络最终层最容易受到局部偏差的影响，部分工作通过将最终层冻结为正交分类器取得了成功。我们受权重冻结导致奇异值恒定的现象启发，对分类器权重进行奇异值分解（SVD）以探究其训练动态。研究发现，在独立同分布（IID）与非独立同分布（non-IID）设置下训练时存在差异。基于此发现，我们提出两种局部训练的正则化项，以持续模拟IID设置：（1）分类器维度级概率分布的方差；（2）编码器表征的超球面均匀性。这些正则化机制促使局部模型无论本地数据分布如何，都能模拟IID环境下的行为，从而在保持数据灵活性的同时抵消偏差倾向。通过在标签偏移与特征偏移两种场景下的大量实验，我们验证了该方法在高度非IID场景中能以显著优势取得最优性能，且对更大模型与数据集具有良好的可扩展性。