Decentralized learning has emerged as an alternative method to the popular parameter-server framework which suffers from high communication burden, single-point failure and scalability issues due to the need of a central server. However, most existing works focus on a single shared model for all workers regardless of the data heterogeneity problem, rendering the resulting model performing poorly on individual workers. In this work, we propose a novel personalized decentralized learning algorithm named DePRL via shared representations. Our algorithm relies on ideas from representation learning theory to learn a low-dimensional global representation collaboratively among all workers in a fully decentralized manner, and a user-specific low-dimensional local head leading to a personalized solution for each worker. We show that DePRL achieves, for the first time, a provable linear speedup for convergence with general non-linear representations (i.e., the convergence rate is improved linearly with respect to the number of workers). Experimental results support our theoretical findings showing the superiority of our method in data heterogeneous environments.

翻译：去中心化学习已成为流行的参数服务器框架的替代方法，后者因依赖中心服务器而面临高通信负担、单点故障和可扩展性问题。然而，现有研究大多关注所有工作节点共享单一模型，未考虑数据异质性，导致模型在单个工作节点上表现不佳。本文提出一种新颖的个性化去中心化学习算法DePRL，通过共享表征实现个性化学习。该算法基于表征学习理论，以完全去中心化的方式在所有工作节点间协作学习低维全局表征，并为每个工作节点生成用户特定的低维局部头（local head），从而提供个性化解决方案。我们首次证明，在通用非线性表征（即收敛速率随工作节点数量线性提升）下，DePRL可实现可证明的线性加速收敛。实验结果验证了理论发现，表明该方法在数据异质环境下具有优越性。