The problem of heterogeneous clients in federated learning has recently drawn a lot of attention. Spectral model sharding, i.e., partitioning the model parameters into low-rank matrices based on the singular value decomposition, has been one of the proposed solutions for more efficient on-device training in such settings. In this work, we present two sampling strategies for such sharding, obtained as solutions to specific optimization problems. The first produces unbiased estimators of the original weights, while the second aims to minimize the squared approximation error. We discuss how both of these estimators can be incorporated in the federated learning loop and practical considerations that arise during local training. Empirically, we demonstrate that both of these methods can lead to improved performance on various commonly used datasets.

翻译：联邦学习中异构客户端的问题近来备受关注。谱模型分片——即基于奇异值分解将模型参数划分为低秩矩阵——已成为此类场景下实现更高效设备端训练的解决方案之一。本文针对此类分片提出了两种采样策略，它们分别是特定优化问题的解。第一种策略生成原始权重的无偏估计量，而第二种策略旨在最小化平方逼近误差。我们讨论了如何将这两种估计量融入联邦学习循环，以及在本地训练过程中出现的实际考量。通过实证研究，我们证明这两种方法均能在多种常用数据集上带来性能提升。