In large-scale federated and decentralized learning, communication efficiency is one of the most challenging bottlenecks. While gossip communication -- where agents can exchange information with their connected neighbors -- is more cost-effective than communicating with the remote server, it often requires a greater number of communication rounds, especially for large and sparse networks. To tackle the trade-off, we examine the communication efficiency under a semi-decentralized communication protocol, in which agents can perform both agent-to-agent and agent-to-server communication in a probabilistic manner. We design a tailored communication-efficient algorithm over semi-decentralized networks, referred to as PISCO, which inherits the robustness to data heterogeneity thanks to gradient tracking and allows multiple local updates for saving communication. We establish the convergence rate of PISCO for nonconvex problems and show that PISCO enjoys a linear speedup in terms of the number of agents and local updates. Our numerical results highlight the superior communication efficiency of PISCO and its resilience to data heterogeneity and various network topologies.

翻译：在大规模联邦学习和去中心化学习中，通信效率是最具挑战性的瓶颈之一。尽管基于gossip的通信方式——智能体可与邻近节点交换信息——相比与远程服务器通信更具成本效益，但在大规模稀疏网络中，该方法通常需要更多通信轮次。为解决这一权衡，我们研究了半去中心化通信协议下的通信效率，该协议允许智能体以概率方式同时进行智能体间通信和智能体与服务器通信。我们为半去中心化网络设计了一种定制化的通信高效算法PISCO，该算法通过梯度追踪机制继承了应对数据异构性的鲁棒性，并允许通过多次本地更新来节省通信开销。我们建立了PISCO在非凸问题上的收敛速率，证明其能实现关于智能体数量和本地更新次数的线性加速。数值实验凸显了PISCO卓越的通信效率，及其对数据异构性和多种网络拓扑结构的适应性。