Decentralized learning strategies allow a collection of agents to learn efficiently from local data sets without the need for central aggregation or orchestration. Current decentralized learning paradigms typically rely on an averaging mechanism to encourage agreement in the parameter space. We argue that in the context of deep neural networks, which are often over-parameterized, encouraging consensus of the neural network outputs, as opposed to their parameters can be more appropriate. This motivates the development of a new decentralized learning algorithm, termed DRT diffusion, based on deep relative trust (DRT), a recently introduced similarity measure for neural networks. We provide convergence analysis for the proposed strategy, and numerically establish its benefit to generalization, especially with sparse topologies, in an image classification task.

翻译：去中心化学习策略使得一组智能体能够从本地数据集中高效学习，而无需中央聚合或协调。当前去中心化学习范式通常依赖于平均机制来促进参数空间的一致性。我们认为，在深度神经网络（通常存在过参数化）的背景下，鼓励神经网络输出（而非其参数）达成共识可能更为合适。这促使我们开发一种新的去中心化学习算法——DRT扩散算法，其基于深度相对信任（DRT）——一种最近提出的神经网络相似性度量。我们为所提策略提供了收敛性分析，并通过图像分类任务中的数值实验，证实了其在提升泛化能力（尤其在稀疏拓扑结构中）的优势。