Robust distributed learning with Byzantine failures has attracted extensive research interests in recent years. However, most of existing methods suffer from curse of dimensionality, which is increasingly serious with the growing complexity of modern machine learning models. In this paper, we design a new method that is suitable for high dimensional problems, under arbitrary number of Byzantine attackers. The core of our design is a direct high dimensional semi-verified mean estimation method. Our idea is to identify a subspace first. The components of mean value perpendicular to this subspace can be estimated via gradient vectors uploaded from worker machines, while the components within this subspace are estimated using auxiliary dataset. We then use our new method as the aggregator of distributed learning problems. Our theoretical analysis shows that the new method has minimax optimal statistical rates. In particular, the dependence on dimensionality is significantly improved compared with previous works.

翻译：近年来，鲁棒的拜占庭故障下的分布式学习引起了广泛的研究兴趣。然而，现有方法大多受限于维度灾难，这一问题随着现代机器学习模型复杂度的日益增长而愈发严重。本文针对任意数目的拜占庭攻击者，提出了一种适用于高维问题的新方法。我们设计的核心是一种直接的高维半验证均值估计方法。其思路为：首先识别一个子空间，垂直于该子空间的均值分量可通过工作机器上传的梯度向量进行估计，而该子空间内的分量则利用辅助数据集进行估计。随后，我们将这一新方法作为分布式学习问题的聚合器。理论分析表明，该方法具有极小化最优的统计速率。特别地，其对于维度的依赖性相较于先前工作得到了显著改善。