We introduce the paradigm of validated decentralized learning for undirected networks with heterogeneous data and possible adversarial infiltration. We require (a) convergence to a global empirical loss minimizer when adversaries are absent, and (b) either detection of adversarial presence of convergence to an admissible consensus irrespective of the adversarial configuration. To this end, we propose the VALID protocol which, to the best of our knowledge, is the first to achieve a validated learning guarantee. Moreover, VALID offers an O(1/T) convergence rate (under pertinent regularity assumptions), and computational and communication complexities comparable to non-adversarial distributed stochastic gradient descent. Remarkably, VALID retains optimal performance metrics in adversary-free environments, sidestepping the robustness penalties observed in prior byzantine-robust methods. A distinctive aspect of our study is a heterogeneity metric based on the norms of individual agents' gradients computed at the global empirical loss minimizer. This not only provides a natural statistic for detecting significant byzantine disruptions but also allows us to prove the optimality of VALID in wide generality. Lastly, our numerical results reveal that, in the absence of adversaries, VALID converges faster than state-of-the-art byzantine robust algorithms, while when adversaries are present, VALID terminates with each honest either converging to an admissible consensus of declaring adversarial presence in the network.

翻译：我们针对存在异构数据与潜在对抗性渗透的无向网络，提出了验证式去中心化学习的范式。我们要求：(a) 在无对抗者时收敛至全局经验损失极小值；(b) 无论对抗性配置如何，要么检测到对抗性存在，要么收敛至可接受的共识。为此，我们提出VALID协议——据我们所知，这是首个实现验证式学习保障的协议。此外，在相关正则性假设下，VALID具有O(1/T)的收敛速率，其计算与通信复杂度与非对抗性分布式随机梯度下降法相当。值得关注的是，VALID在无对抗性环境中仍能保持最优性能指标，规避了先前的拜占庭鲁棒方法中存在的鲁棒性惩罚问题。本研究的独特之处在于基于全局经验损失极小值处各智能体梯度范数构建的异构性度量指标。该指标既可提供检测重大拜占庭干扰的自然统计量，又能证明VALID在广泛场景下的最优性。最后，数值实验表明：无对抗者时，VALID的收敛速度快于最先进的拜占庭鲁棒算法；存在对抗者时，VALID会终止运行，此时每个诚实节点要么收敛至可接受的共识，要么声明网络中检测到对抗性存在。