Batch normalization has proven to be a very beneficial mechanism to accelerate the training and improve the accuracy of deep neural networks in centralized environments. Yet, the scheme faces significant challenges in federated learning, especially under high data heterogeneity. Essentially, the main challenges arise from external covariate shifts and inconsistent statistics across clients. We introduce in this paper Federated BatchNorm (FBN), a novel scheme that restores the benefits of batch normalization in federated learning. Essentially, FBN ensures that the batch normalization during training is consistent with what would be achieved in a centralized execution, hence preserving the distribution of the data, and providing running statistics that accurately approximate the global statistics. FBN thereby reduces the external covariate shift and matches the evaluation performance of the centralized setting. We also show that, with a slight increase in complexity, we can robustify FBN to mitigate erroneous statistics and potentially adversarial attacks.

翻译：批量归一化已被证明是一种非常有效的机制，能够加速深度神经网络在集中式环境中的训练并提高其准确性。然而，该方案在联邦学习中，尤其是在数据高度异构的情况下，面临着重大挑战。本质上，主要挑战源于外部协变量偏移和跨客户端统计量不一致。本文中，我们引入了联邦批量归一化，这是一种新颖的方案，旨在恢复批量归一化在联邦学习中的优势。本质上，FBN 确保训练期间的批量归一化与集中式执行所能实现的效果保持一致，从而保留数据的分布特性，并提供能够准确近似全局统计量的运行统计量。因此，FBN 减少了外部协变量偏移，并匹配了集中式设置下的评估性能。我们还表明，通过略微增加复杂度，我们可以增强 FBN 的鲁棒性，以减轻错误统计量和潜在对抗性攻击的影响。