Differential privacy is a widely accepted measure of privacy in the context of deep learning algorithms, and achieving it relies on a noisy training approach known as differentially private stochastic gradient descent (DP-SGD). DP-SGD requires direct noise addition to every gradient in a dense neural network, the privacy is achieved at a significant utility cost. In this work, we present Spectral-DP, a new differentially private learning approach which combines gradient perturbation in the spectral domain with spectral filtering to achieve a desired privacy guarantee with a lower noise scale and thus better utility. We develop differentially private deep learning methods based on Spectral-DP for architectures that contain both convolution and fully connected layers. In particular, for fully connected layers, we combine a block-circulant based spatial restructuring with Spectral-DP to achieve better utility. Through comprehensive experiments, we study and provide guidelines to implement Spectral-DP deep learning on benchmark datasets. In comparison with state-of-the-art DP-SGD based approaches, Spectral-DP is shown to have uniformly better utility performance in both training from scratch and transfer learning settings.

翻译：差分隐私是深度学习算法中被广泛接受的隐私度量标准，其实现依赖于一种称为差分隐私随机梯度下降（DP-SGD）的噪声训练方法。DP-SGD需要对稠密神经网络中的每个梯度直接添加噪声，这种隐私保障以显著的效用损失为代价。本文提出了一种新的差分隐私学习方法——Spectral-DP，该方法将谱域中的梯度扰动与谱滤波相结合，以较低的噪声规模实现所需的隐私保障，从而获得更好的效用。我们基于Spectral-DP为包含卷积层和全连接层的架构开发了差分隐私深度学习方法。特别地，对于全连接层，我们将基于块循环矩阵的空间重构与Spectral-DP相结合，以获得更好的效用。通过全面的实验，我们研究并提供了在基准数据集上实现Spectral-DP深度学习的指南。与基于DP-SGD的先进方法相比，Spectral-DP在从头训练和迁移学习两种场景下均表现出更优的效用性能。