In large-scale federated and decentralized learning, communication efficiency is one of the most challenging bottlenecks. While gossip communication -- where agents can exchange information with their connected neighbors -- is more cost-effective than communicating with the remote server, it often requires a greater number of communication rounds, especially for large and sparse networks. To tackle the trade-off, we examine the communication efficiency under a semi-decentralized communication protocol, in which agents can perform both agent-to-agent and agent-to-server communication in a probabilistic manner. We design a tailored communication-efficient algorithm over semi-decentralized networks, referred to as PISCO, which inherits the robustness to data heterogeneity thanks to gradient tracking and allows multiple local updates for saving communication. We establish the convergence rate of PISCO for nonconvex problems and show that PISCO enjoys a linear speedup in terms of the number of agents and local updates. Our numerical results highlight the superior communication efficiency of PISCO and its resilience to data heterogeneity and various network topologies.

翻译：在大规模联邦与去中心化学习中，通信效率是最具挑战性的瓶颈之一。虽然代理节点可与相连邻居进行信息交换的流言通信模式比与远程服务器通信更具成本效益，但其通常需要更多通信轮数，尤其在大型稀疏网络中更为显著。为权衡这一矛盾，我们研究了半去中心化通信协议下的通信效率，该协议允许代理节点以概率化方式同时执行代理间通信与代理-服务器通信。我们设计了一种面向半去中心化网络的定制化通信高效算法PISCO，该算法通过梯度跟踪机制保持对数据异质性的鲁棒性，并支持多次本地更新以节约通信开销。我们建立了PISCO在非凸问题上的收敛速率，证明其在代理数量与本地更新次数方面具有线性加速特性。数值实验结果凸显了PISCO卓越的通信效率及其对数据异质性与多种网络拓扑的强适应能力。