Differentially private (stochastic) gradient descent is the workhorse of DP private machine learning in both the convex and non-convex settings. Without privacy constraints, second-order methods, like Newton's method, converge faster than first-order methods like gradient descent. In this work, we investigate the prospect of using the second-order information from the loss function to accelerate DP convex optimization. We first develop a private variant of the regularized cubic Newton method of Nesterov and Polyak, and show that for the class of strongly convex loss functions, our algorithm has quadratic convergence and achieves the optimal excess loss. We then design a practical second-order DP algorithm for the unconstrained logistic regression problem. We theoretically and empirically study the performance of our algorithm. Empirical results show our algorithm consistently achieves the best excess loss compared to other baselines and is 10-40x faster than DP-GD/DP-SGD.

翻译：差分隐私（随机）梯度下降是凸与非凸场景下差分隐私机器学习的主要工作手段。在没有隐私约束的情况下，牛顿法等二阶方法的收敛速度快于梯度下降等一阶方法。在本工作中，我们探讨利用损失函数的二阶信息加速差分隐私凸优化的可能性。我们首先开发了Nesterov和Polyak正则化三次牛顿方法的隐私变体，并证明对于强凸损失函数类，我们的算法具有二次收敛性且达到最优超额损失。随后，我们为无约束逻辑回归问题设计了一种实用的二阶差分隐私算法。我们从理论和实证两个层面研究算法性能。实验结果表明，与其他基线方法相比，我们的算法始终能实现最佳超额损失，且速度比DP-GD/DP-SGD快10-40倍。