Machine learning models are known to memorize private data to reduce their training loss, which can be inadvertently exploited by privacy attacks such as model inversion and membership inference. To protect against these attacks, differential privacy (DP) has become the de facto standard for privacy-preserving machine learning, particularly those popular training algorithms using stochastic gradient descent, such as DPSGD. Nonetheless, DPSGD still suffers from severe utility loss due to its slow convergence. This is partially caused by the random sampling, which brings bias and variance to the gradient, and partially by the Gaussian noise, which leads to fluctuation of gradient updates. Our key idea to address these issues is to apply selective updates to the model training, while discarding those useless or even harmful updates. Motivated by this, this paper proposes DPSUR, a Differentially Private training framework based on Selective Updates and Release, where the gradient from each iteration is evaluated based on a validation test, and only those updates leading to convergence are applied to the model. As such, DPSUR ensures the training in the right direction and thus can achieve faster convergence than DPSGD. The main challenges lie in two aspects -- privacy concerns arising from gradient evaluation, and gradient selection strategy for model update. To address the challenges, DPSUR introduces a clipping strategy for update randomization and a threshold mechanism for gradient selection. Experiments conducted on MNIST, FMNIST, CIFAR-10, and IMDB datasets show that DPSUR significantly outperforms previous works in terms of convergence speed and model utility.

翻译：机器学习模型已知会通过记忆私有数据来降低训练损失，这一特性可能被模型反转攻击和成员推断攻击等隐私攻击无意中利用。为抵御此类攻击，差分隐私已成为隐私保护机器学习的事实标准，尤其适用于随机梯度下降这类主流训练算法（如DPSGD）。然而，DPSGD仍因收敛缓慢而面临严重的效用损失，其部分原因在于随机采样带来的梯度偏差与方差，部分原因在于高斯噪声导致的梯度更新波动。解决这些问题的核心思路是对模型训练采用选择性更新，同时舍弃那些无用甚至有害的更新。受此启发，本文提出DPSUR——一种基于选择性更新与发布的差分隐私训练框架。该框架通过验证测试评估每次迭代的梯度，仅将促进收敛的更新应用于模型。因此，DPSUR能确保训练沿正确方向进行，从而比DPSGD实现更快的收敛速度。主要挑战来自两方面：梯度评估引发的隐私问题，以及模型更新的梯度选择策略。为应对这些挑战，DPSUR引入了更新随机化的裁剪策略与梯度选择的阈值机制。在MNIST、FMNIST、CIFAR-10和IMDB数据集上的实验表明，DPSUR在收敛速度和模型效用方面显著优于现有方法。