In large-scale federated and decentralized learning, communication efficiency is one of the most challenging bottlenecks. While gossip communication -- where agents can exchange information with their connected neighbors -- is more cost-effective than communicating with the remote server, it often requires a greater number of communication rounds, especially for large and sparse networks. To tackle the trade-off, we examine the communication efficiency under a semi-decentralized communication protocol, in which agents can perform both agent-to-agent and agent-to-server communication in a probabilistic manner. We design a tailored communication-efficient algorithm over semi-decentralized networks, referred to as PISCO, which inherits the robustness to data heterogeneity thanks to gradient tracking and allows multiple local updates for saving communication. We establish the convergence rate of PISCO for nonconvex problems and show that PISCO enjoys a linear speedup in terms of the number of agents and local updates. Our numerical results highlight the superior communication efficiency of PISCO and its resilience to data heterogeneity and various network topologies.

翻译：在大规模联邦与去中心化学习中，通信效率是最具挑战性的瓶颈之一。尽管点对点通信（智能体可与连接邻居交换信息）相比远程服务器通信更具成本效益，但通常需要更多通信轮次，尤其在大型稀疏网络中尤为明显。为应对这一权衡问题，我们研究了半去中心化通信协议下的通信效率，该协议允许智能体以概率方式同时进行智能体间与智能体-服务器通信。我们设计了面向半去中心化网络的定制化通信高效算法PISCO，该算法因采用梯度追踪技术而继承了对数据异质性的鲁棒性，并允许通过多次本地更新来节省通信开销。我们建立了非凸问题下PISCO的收敛速率，证明其收敛速度随智能体数量及本地更新次数呈线性加速。数值实验结果突显了PISCO卓越的通信效率，及其对数据异质性与多种网络拓扑结构的适应能力。