The theory underlying robust distributed learning algorithms, designed to resist adversarial machines, matches empirical observations when data is homogeneous. Under data heterogeneity however, which is the norm in practical scenarios, established lower bounds on the learning error are essentially vacuous and greatly mismatch empirical observations. This is because the heterogeneity model considered is too restrictive and does not cover basic learning tasks such as least-squares regression. We consider in this paper a more realistic heterogeneity model, namely (G,B)-gradient dissimilarity, and show that it covers a larger class of learning problems than existing theory. Notably, we show that the breakdown point under heterogeneity is lower than the classical fraction 1/2. We also prove a new lower bound on the learning error of any distributed learning algorithm. We derive a matching upper bound for a robust variant of distributed gradient descent, and empirically show that our analysis reduces the gap between theory and practice.

翻译：支撑鲁棒分布式学习算法的理论（旨在抵御对抗性机器）在数据同质性下与经验观察一致。然而，在数据异质性（实际场景中的常态）下，已建立的关于学习误差的下界本质上是空洞的，且与经验观察严重不符。这是因为所考虑的异质性模型过于局限，并未涵盖诸如最小二乘回归等基本学习任务。本文考虑了一种更符合实际的异质性模型，即(G,B)-梯度差异性，并证明该模型比现有理论涵盖了更广泛的学习问题类。值得注意的是，我们证明了异质性下的崩溃点低于经典的1/2分数值。我们还为任何分布式学习算法的学习误差证明了一个新的下界。针对分布式梯度下降的一个鲁棒变体，我们推导出匹配的上界，并通过实验表明，我们的分析缩小了理论与实际之间的差距。