In this paper, we study differentially private empirical risk minimization (DP-ERM). It has been shown that the worst-case utility of DP-ERM reduces polynomially as the dimension increases. This is a major obstacle to privately learning large machine learning models. In high dimension, it is common for some model's parameters to carry more information than others. To exploit this, we propose a differentially private greedy coordinate descent (DP-GCD) algorithm. At each iteration, DP-GCD privately performs a coordinate-wise gradient step along the gradients' (approximately) greatest entry. We show theoretically that DP-GCD can achieve a logarithmic dependence on the dimension for a wide range of problems by naturally exploiting their structural properties (such as quasi-sparse solutions). We illustrate this behavior numerically, both on synthetic and real datasets.

翻译：本文研究差分隐私经验风险最小化（DP-ERM）。已有研究表明，DP-ERM的最坏情况效用随维度增加呈多项式下降，这成为私有化学习大规模机器学习模型的主要障碍。在高维场景中，部分模型参数往往携带更多信息。为利用这一特性，我们提出一种差分隐私贪心坐标下降（DP-GCD）算法。在每次迭代中，DP-GCD沿梯度最大（近似）分量方向私有化执行坐标梯度步。理论分析表明，DP-GCD能够通过自然利用问题的结构特性（如准稀疏解），在广泛问题中实现维度的对数依赖。我们通过合成数据集和真实数据集上的数值实验验证了该特性。