This paper presents a holistic approach to gradient leakage resilient distributed Stochastic Gradient Descent (SGD). First, we analyze two types of strategies for privacy-enhanced federated learning: (i) gradient pruning with random selection or low-rank filtering and (ii) gradient perturbation with additive random noise or differential privacy noise. We analyze the inherent limitations of these approaches and their underlying impact on privacy guarantee, model accuracy, and attack resilience. Next, we present a gradient leakage resilient approach to securing distributed SGD in federated learning, with differential privacy controlled noise as the tool. Unlike conventional methods with the per-client federated noise injection and fixed noise parameter strategy, our approach keeps track of the trend of per-example gradient updates. It makes adaptive noise injection closely aligned throughout the federated model training. Finally, we provide an empirical privacy analysis on the privacy guarantee, model utility, and attack resilience of the proposed approach. Extensive evaluation using five benchmark datasets demonstrates that our gradient leakage resilient approach can outperform the state-of-the-art methods with competitive accuracy performance, strong differential privacy guarantee, and high resilience against gradient leakage attacks. The code associated with this paper can be found: https://github.com/git-disl/Fed-alphaCDP.

翻译：本文提出了一种应对梯度泄露威胁的分布式随机梯度下降（SGD）综合防护方法。首先，我们分析了两种隐私增强型联邦学习策略：（i）基于随机选择或低秩过滤的梯度剪枝方法，以及（ii）结合加性随机噪声或差分隐私噪声的梯度扰动方法。我们深入剖析了这些方法的固有局限性及其对隐私保障、模型精度和攻击抵御能力的潜在影响。其次，我们提出了一种基于差分隐私可控噪声的梯度泄露防御方案，用于保护联邦学习中的分布式SGD过程。不同于传统方法中采用固定噪声参数的逐客户端联邦噪声注入策略，本方法通过追踪逐样本梯度更新的变化趋势，实现了与联邦模型训练过程紧密协同的自适应噪声注入。最后，我们从隐私保障、模型效用和攻击抵御能力三个维度对所提方法进行了实证隐私分析。在五个基准数据集上的大量实验表明，本梯度泄露防御方法在保持竞争力精度的同时，兼具强大的差分隐私保障和优异的梯度泄露攻击抵御能力，性能全面超越现有最优方案。本文相关代码已开源：https://github.com/git-disl/Fed-alphaCDP