Modern machine learning algorithms aim to extract fine-grained information from data to provide accurate predictions, which often conflicts with the goal of privacy protection. This paper addresses the practical and theoretical importance of developing privacy-preserving machine learning algorithms that ensure good performance while preserving privacy. In this paper, we focus on the privacy and utility (measured by excess risk bounds) performances of differentially private stochastic gradient descent (SGD) algorithms in the setting of stochastic convex optimization. Specifically, we examine the pointwise problem in the low-noise setting for which we derive sharper excess risk bounds for the differentially private SGD algorithm. In the pairwise learning setting, we propose a simple differentially private SGD algorithm based on gradient perturbation. Furthermore, we develop novel utility bounds for the proposed algorithm, proving that it achieves optimal excess risk rates even for non-smooth losses. Notably, we establish fast learning rates for privacy-preserving pairwise learning under the low-noise condition, which is the first of its kind.

翻译：现代机器学习算法旨在从数据中提取细粒度信息以提供准确预测，这通常与隐私保护目标相冲突。本文从理论与实践角度阐述了开发兼具良好性能与隐私保护能力的机器学习算法的重要性。我们聚焦于随机凸优化场景下差分隐私随机梯度下降算法的隐私性与效用性（以超额风险界衡量）。具体而言，我们研究了低噪声设定下的逐点问题，为差分隐私SGD算法推导出更紧致的超额风险界。在成对学习场景中，我们提出了一种基于梯度扰动的简单差分隐私SGD算法。进一步，我们为该算法建立了新颖的效用界，证明其即使在非光滑损失函数下也能实现最优超额风险率。值得注意的是，我们首次在低噪声条件下实现了隐私保护成对学习的快速学习率。