Privacy in AI remains a topic that draws attention from researchers and the general public in recent years. As one way to implement privacy-preserving AI, differentially private learning is a framework that enables AI models to use differential privacy (DP). To achieve DP in the learning process, existing algorithms typically limit the magnitude of gradients with a constant clipping, which requires carefully tuned due to its significant impact on model performance. As a solution to this issue, latest works NSGD and Auto-S innovatively propose to use normalization instead of clipping to avoid hyperparameter tuning. However, normalization-based approaches like NSGD and Auto-S rely on a monotonic weight function, which imposes excessive weight on small gradient samples and introduces extra deviation to the update. In this paper, we propose a Differentially Private Per-Sample Adaptive Clipping (DP-PSAC) algorithm based on a non-monotonic adaptive weight function, which guarantees privacy without the typical hyperparameter tuning process of using a constant clipping while significantly reducing the deviation between the update and true batch-averaged gradient. We provide a rigorous theoretical convergence analysis and show that with convergence rate at the same order, the proposed algorithm achieves a lower non-vanishing bound, which is maintained over training iterations, compared with NSGD/Auto-S. In addition, through extensive experimental evaluation, we show that DP-PSAC outperforms or matches the state-of-the-art methods on multiple main-stream vision and language tasks.

翻译：人工智能领域的隐私问题近年来持续受到研究人员和公众的关注。作为实现隐私保护人工智能的一种方式，差分隐私学习是一种使人工智能模型能够运用差分隐私的框架。在学习过程中实现差分隐私时，现有算法通常采用常量裁剪来限制梯度幅度，但由于其对模型性能影响显著，需要精细调整。针对这一问题，最新研究NSGD和Auto-S创新性地提出使用归一化替代裁剪以避免超参数调优。然而，基于归一化的方法（如NSGD和Auto-S）依赖单调权重函数，这会对小梯度样本施加过大权重，并引入额外更新偏差。本文提出一种基于非单调自适应权重函数的差分隐私每样本自适应裁剪算法，该算法在保证隐私性的同时，无需常量裁剪所需的典型超参数调优过程，并显著降低了更新与真实批量平均梯度之间的偏差。我们给出了严格的理论收敛性分析，证明在相同阶数的收敛速度下，与NSGD/Auto-S相比，所提算法在训练迭代中维持更低的非消失界。此外，通过大量实验评估，我们证明DP-PSAC在多个主流视觉与语言任务中达到或超越了现有最优方法的性能。