Federated Learning (FL) is an emerging collaborative machine learning framework where multiple clients train the global model without sharing their own datasets. In FL, the model inconsistency caused by the local data heterogeneity across clients results in the near-orthogonality of client updates, which leads to the global update norm reduction and slows down the convergence. Most previous works focus on eliminating the difference of parameters (or gradients) between the local and global models, which may fail to reflect the model inconsistency due to the complex structure of the machine learning model and the Euclidean space's limitation in meaningful geometric representations. In this paper, we propose FedMRUR by adopting the manifold model fusion scheme and a new global optimizer to alleviate the negative impacts. Concretely, FedMRUR adopts a hyperbolic graph manifold regularizer enforcing the representations of the data in the local and global models are close to each other in a low-dimensional subspace. Because the machine learning model has the graph structure, the distance in hyperbolic space can reflect the model bias better than the Euclidean distance. In this way, FedMRUR exploits the manifold structures of the representations to significantly reduce the model inconsistency. FedMRUR also aggregates the client updates norms as the global update norm, which can appropriately enlarge each client's contribution to the global update, thereby mitigating the norm reduction introduced by the near-orthogonality of client updates. Furthermore, we theoretically prove that our algorithm can achieve a linear speedup property for non-convex setting under partial client participation.Experiments demonstrate that FedMRUR can achieve a new state-of-the-art (SOTA) accuracy with less communication.

翻译：联邦学习（FL）是一种新兴的协作式机器学习框架，其中多个客户端在不共享各自数据集的情况下训练全局模型。在FL中，由客户端本地数据异质性导致的模型不一致性会引发客户端更新的近似正交性，从而造成全局更新范数缩减并减慢收敛速度。以往多数研究侧重于消除局部模型与全局模型之间参数（或梯度）的差异，但受限于机器学习模型的复杂结构及欧氏空间在有效几何表征方面的局限性，这类方法难以真实反映模型不一致性。本文提出FedMRUR方法，通过采用流形模型融合方案与新型全局优化器来缓解上述负面影响。具体而言，FedMRUR引入双曲图流形正则化器，强制局部模型与全局模型中的数据表征在低维子空间中保持接近。由于机器学习模型具有图结构特性，双曲空间中的距离比欧氏距离更能反映模型偏差。通过这种方式，FedMRUR利用表征的流形结构显著降低模型不一致性。同时，FedMRUR将客户端更新范数聚合为全局更新范数，从而合理扩大每个客户端对全局更新的贡献，有效缓解因客户端更新近似正交性导致的范数缩减问题。此外，我们从理论上证明，在部分客户端参与的非凸场景下，该算法可实现线性加速特性。实验表明，FedMRUR能以更少通信量达到新的最优（SOTA）精度。