The theory underlying robust distributed learning algorithms, designed to resist adversarial machines, matches empirical observations when data is homogeneous. Under data heterogeneity however, which is the norm in practical scenarios, established lower bounds on the learning error are essentially vacuous and greatly mismatch empirical observations. This is because the heterogeneity model considered is too restrictive and does not cover basic learning tasks such as least-squares regression. We consider in this paper a more realistic heterogeneity model, namely (G,B)-gradient dissimilarity, and show that it covers a larger class of learning problems than existing theory. Notably, we show that the breakdown point under heterogeneity is lower than the classical fraction 1/2. We also prove a new lower bound on the learning error of any distributed learning algorithm. We derive a matching upper bound for a robust variant of distributed gradient descent, and empirically show that our analysis reduces the gap between theory and practice.

翻译：鲁棒分布式学习算法旨在抵御恶意节点，其理论基础在数据同质环境下与实验观察相符。然而在数据异质性（这在实际情况中通常占据主导地位）条件下，现有关于学习误差的下界本质上缺乏实际意义，与实验观察严重不符。这是由于所考虑的异质性模型过于局限，无法涵盖诸如最小二乘回归等基础学习任务。本文提出一种更为实际的异质性模型，即（G,B）-梯度差异度量，并证明该模型能够覆盖比现有理论更广泛的学习问题。值得注意的是，我们证明异质性环境下的崩溃点低于传统1/2阈值。我们同时证明了任意分布式学习算法在学习误差上的新下界。针对分布式梯度下降的鲁棒变体算法，我们推导了相匹配的上界，并通过实验证明我们的分析有效缩小了理论与实践的差距。