Robust distributed learning with Byzantine failures has attracted extensive research interests in recent years. However, most of existing methods suffer from curse of dimensionality, which is increasingly serious with the growing complexity of modern machine learning models. In this paper, we design a new method that is suitable for high dimensional problems, under arbitrary number of Byzantine attackers. The core of our design is a direct high dimensional semi-verified mean estimation method. Our idea is to identify a subspace first. The components of mean value perpendicular to this subspace can be estimated via gradient vectors uploaded from worker machines, while the components within this subspace are estimated using auxiliary dataset. We then use our new method as the aggregator of distributed learning problems. Our theoretical analysis shows that the new method has minimax optimal statistical rates. In particular, the dependence on dimensionality is significantly improved compared with previous works.

翻译：近年来，鲁棒的分布式学习在拜占庭故障场景下引起了广泛研究兴趣。然而，现有大多数方法受限于维度灾难问题，随着现代机器学习模型日益复杂，这一挑战愈发严峻。本文针对高维问题，设计了一种能容忍任意数量拜占庭攻击者的新方法。其核心创新在于提出了一种直接的高维半验证均值估计方法。基本思路是：首先识别一个子空间，其中垂直于该子空间的均值分量可通过工作节点上传的梯度向量进行估计，而该子空间内的分量则利用辅助数据集进行估计。随后，我们将该新方法作为分布式学习问题的聚合器。理论分析表明，新方法达到了极小极大最优统计速率。特别地，与现有工作相比，方法对维度的依赖性得到了显著改善。