Gradient inversion attacks are an ubiquitous threat in federated learning as they exploit gradient leakage to reconstruct supposedly private training data. Recent work has proposed to prevent gradient leakage without loss of model utility by incorporating a PRivacy EnhanCing mODulE (PRECODE) based on variational modeling. Without further analysis, it was shown that PRECODE successfully protects against gradient inversion attacks. In this paper, we make multiple contributions. First, we investigate the effect of PRECODE on gradient inversion attacks to reveal its underlying working principle. We show that variational modeling introduces stochasticity into the gradients of PRECODE and the subsequent layers in a neural network. The stochastic gradients of these layers prevent iterative gradient inversion attacks from converging. Second, we formulate an attack that disables the privacy preserving effect of PRECODE by purposefully omitting stochastic gradients during attack optimization. To preserve the privacy preserving effect of PRECODE, our analysis reveals that variational modeling must be placed early in the network. However, early placement of PRECODE is typically not feasible due to reduced model utility and the exploding number of additional model parameters. Therefore, as a third contribution, we propose a novel privacy module -- the Convolutional Variational Bottleneck (CVB) -- that can be placed early in a neural network without suffering from these drawbacks. We conduct an extensive empirical study on three seminal model architectures and six image classification datasets. We find that all architectures are susceptible to gradient leakage attacks, which can be prevented by our proposed CVB. Compared to PRECODE, we show that our novel privacy module requires fewer trainable parameters, and thus computational and communication costs, to effectively preserve privacy.

翻译：梯度反演攻击是联邦学习中普遍存在的威胁，它利用梯度泄漏重建理应保密的训练数据。近期研究提出通过引入基于变分建模的隐私增强模块（PRECODE），在不损失模型效用的前提下防止梯度泄漏。此前分析已表明PRECODE能有效抵御梯度反演攻击。本文做出多项贡献：首先，我们探究PRECODE对梯度反演攻击的影响，揭示其潜在工作原理。研究表明，变分建模将随机性引入PRECODE及神经网络后续层的梯度中，这些层的随机梯度阻止了迭代梯度反演攻击的收敛。其次，我们设计了一种攻击方法，通过在攻击优化过程中刻意忽略随机梯度，从而消除PRECODE的隐私保护效果。分析表明，为保持PRECODE的隐私保护效果，变分建模必须置于网络早期位置。然而，由于模型效用下降及额外模型参数数量激增，PRECODE的早期部署通常不可行。因此，作为第三项贡献，我们提出新型隐私模块——卷积变分瓶颈（CVB），该模块可在神经网络早期部署且避免上述缺陷。我们在三种经典模型架构和六个图像分类数据集上进行了广泛实证研究，发现所有架构均易受梯度泄漏攻击，而本文提出的CVB能有效防止此类攻击。与PRECODE相比，我们的新型隐私模块需要更少可训练参数，从而以更低计算和通信成本实现有效隐私保护。