In the paradigm of decentralized learning, a group of agents collaborate to learn a global model using a distributed dataset without a central server; nevertheless, it is severely challenged by the heterogeneity of the data distribution across the agents. For example, the data may be distributed non-independently and identically, and even be noised or poisoned. To address these data challenges, we propose ROSS, a novel robust decentralized stochastic learning algorithm based on Shapley values, in this paper. Specifically, in each round, each agent aggregates the cross-gradient information from its neighbors, i.e., the derivatives of its local model with respect to the datasets of its neighbors, to update its local model in a momentum like manner, while we innovate in weighting the derivatives according to their contributions measured by Shapley values. We perform solid theoretical analysis to reveal the linear convergence speedup of our ROSS algorithm. We also verify the efficacy of our algorithm through extensive experiments on public datasets. Our results demonstrate that, in face of the above variety of data challenges, our ROSS algorithm has significant advantages over existing state-of-the-art proposals in terms of both convergence and prediction accuracy.

翻译：在去中心化学习范式中，一组智能体利用分布式数据集协作学习全局模型而无需中央服务器；然而，跨智能体的数据分布异质性对此提出了严峻挑战。例如，数据可能以非独立同分布形式存在，甚至包含噪声或被恶意投毒。为应对这些数据挑战，本文提出ROSS——一种基于沙普利值的新型鲁棒去中心化随机学习算法。具体而言，在每轮迭代中，每个智能体聚合来自其邻居的交叉梯度信息（即其局部模型对邻居数据集的导数），以类动量方式更新其局部模型；我们的创新在于根据沙普利值度量的贡献度对这些导数进行加权。我们通过严谨的理论分析揭示了ROSS算法的线性收敛加速特性，并在公开数据集上通过大量实验验证了算法的有效性。结果表明，在面对上述各类数据挑战时，ROSS算法在收敛速度和预测精度方面均显著优于现有最先进方案。