DP-SGD has emerged as a popular method to protect personally identifiable information in deep learning applications. Unfortunately, DP-SGD's per-sample gradient clipping and uniform noise addition during training can significantly degrade model utility. To enhance the model's utility, researchers proposed various adaptive DP-SGD methods. However, we examine and discover that these techniques result in greater privacy leakage or lower accuracy than the traditional DP-SGD method, or a lack of evaluation on a complex data set such as CIFAR100. To address these limitations, we propose an Auto DP-SGD. Our method automates clipping threshold estimation based on the DL model's gradient norm and scales the gradients of each training sample without losing gradient information. This helps to improve the algorithm's utility while using a less privacy budget. To further improve accuracy, we introduce automatic noise multiplier decay mechanisms to decrease the noise multiplier after every epoch. Finally, we develop closed-form mathematical expressions using tCDP accountant for automatic noise multiplier and automatic clipping threshold estimation. Through extensive experimentation, we demonstrate that Auto DP-SGD outperforms existing SOTA DP-SGD methods in privacy and accuracy on various benchmark datasets. We also show that privacy can be improved by lowering the scale factor and using learning rate schedulers without significantly reducing accuracy. Specifically, Auto DP-SGD, when used with a step noise multiplier, improves accuracy by 3.20, 1.57, 6.73, and 1.42 for the MNIST, CIFAR10, CIFAR100, and AG News Corpus datasets, respectively. Furthermore, it obtains a substantial reduction in the privacy budget of 94.9, 79.16, 67.36, and 53.37 for the corresponding data sets.

翻译：DP-SGD已成为深度学习中保护个人可识别信息的流行方法。然而，DP-SGD训练过程中的逐样本梯度裁剪与均匀噪声添加会显著降低模型效用。为提升模型效用，研究者提出了多种自适应DP-SGD方法。但我们经检验发现，这些技术会导致比传统DP-SGD方法更严重的隐私泄露或更低的准确率，或缺乏对CIFAR100等复杂数据集的评估。针对上述局限，我们提出Auto DP-SGD方法。该方法基于深度学习模型的梯度范数自动估计裁剪阈值，并在不损失梯度信息的前提下缩放每个训练样本的梯度，从而在消耗更少隐私预算的同时提升算法效用。为进一步提高准确率，我们引入自动噪声乘子衰减机制，每轮训练后递减噪声乘子。最后，我们利用tCDP会计方法推导出用于自动噪声乘子与自动裁剪阈值估计的闭式数学表达式。通过大量实验证明，Auto DP-SGD在多个基准数据集上的隐私性与准确性均优于现有最先进的DP-SGD方法。我们还表明，通过降低缩放因子并结合学习率调度器可在不显著降低准确率的前提下提升隐私保护效果。具体而言，采用阶梯式噪声乘子的Auto DP-SGD在MNIST、CIFAR10、CIFAR100和AG News Corpus数据集上分别将准确率提升了3.20、1.57、6.73和1.42个百分点，同时对应数据集的隐私预算分别大幅降低了94.9、79.16、67.36和53.37。