In Byzantine robust distributed or federated learning, a central server wants to train a machine learning model over data distributed across multiple workers. However, a fraction of these workers may deviate from the prescribed algorithm and send arbitrary messages. While this problem has received significant attention recently, most current defenses assume that the workers have identical data. For realistic cases when the data across workers are heterogeneous (non-iid), we design new attacks which circumvent current defenses, leading to significant loss of performance. We then propose a simple bucketing scheme that adapts existing robust algorithms to heterogeneous datasets at a negligible computational cost. We also theoretically and experimentally validate our approach, showing that combining bucketing with existing robust algorithms is effective against challenging attacks. Our work is the first to establish guaranteed convergence for the non-iid Byzantine robust problem under realistic assumptions.

翻译：在拜占庭鲁棒分布式或联邦学习中，中央服务器希望基于分布在多个工作节点上的数据训练机器学习模型。然而，部分工作节点可能偏离规定的算法并发送任意消息。尽管该问题近期受到广泛关注，但现有防御措施多数假设各工作节点持有相同数据。针对工作节点数据异构（非独立同分布）的现实场景，我们设计了能够规避当前防御机制的新型攻击，导致模型性能显著下降。随后，我们提出一种简单的分桶方案，以可忽略的计算成本使现有鲁棒算法适应异构数据集。我们通过理论与实验验证了该方法，表明将分桶与现有鲁棒算法相结合能有效抵御具有挑战性的攻击。本研究首次在现实假设下为非独立同分布拜占庭鲁棒问题建立了收敛性保证。