Differentially private SGD (DP-SGD) holds the promise of enabling the safe and responsible application of machine learning to sensitive datasets. However, DP-SGD only provides a biased, noisy estimate of a mini-batch gradient. This renders optimisation steps less effective and limits model utility as a result. With this work, we show a connection between per-sample gradient norms and the estimation bias of the private gradient oracle used in DP-SGD. Here, we propose Bias-Aware Minimisation (BAM) that allows for the provable reduction of private gradient estimator bias. We show how to efficiently compute quantities needed for BAM to scale to large neural networks and highlight similarities to closely related methods such as Sharpness-Aware Minimisation. Finally, we provide empirical evidence that BAM not only reduces bias but also substantially improves privacy-utility trade-offs on the CIFAR-10, CIFAR-100, and ImageNet-32 datasets.

翻译：差分私有SGD（DP-SGD）有望实现机器学习对敏感数据集的安全与负责任应用。然而，DP-SGD仅能提供小批量梯度的有偏噪声估计，这降低了优化步骤的有效性，并因此限制了模型效用。本研究揭示了每个样本梯度范数与DP-SGD中使用的私有梯度估计器偏差之间的关联。我们提出偏差感知最小化（BAM）方法，可证明地减少私有梯度估计器的偏差。我们展示了如何高效计算BAM所需量值以扩展至大型神经网络，并强调其与锐度感知最小化等密切相关方法的相似性。最后，我们提供实证证据表明，BAM不仅减少了偏差，还在CIFAR-10、CIFAR-100和ImageNet-32数据集上显著改善了隐私-效用权衡。