Differential private optimization for nonconvex smooth objective is considered. In the previous work, the best known utility bound is $\widetilde O(\sqrt{d}/(n\varepsilon_\mathrm{DP}))$ in terms of the squared full gradient norm, which is achieved by Differential Private Gradient Descent (DP-GD) as an instance, where $n$ is the sample size, $d$ is the problem dimensionality and $\varepsilon_\mathrm{DP}$ is the differential privacy parameter. To improve the best known utility bound, we propose a new differential private optimization framework called \emph{DIFF2 (DIFFerential private optimization via gradient DIFFerences)} that constructs a differential private global gradient estimator with possibly quite small variance based on communicated \emph{gradient differences} rather than gradients themselves. It is shown that DIFF2 with a gradient descent subroutine achieves the utility of $\widetilde O(d^{2/3}/(n\varepsilon_\mathrm{DP})^{4/3})$, which can be significantly better than the previous one in terms of the dependence on the sample size $n$. To the best of our knowledge, this is the first fundamental result to improve the standard utility $\widetilde O(\sqrt{d}/(n\varepsilon_\mathrm{DP}))$ for nonconvex objectives. Additionally, a more computational and communication efficient subroutine is combined with DIFF2 and its theoretical analysis is also given. Numerical experiments are conducted to validate the superiority of DIFF2 framework.

翻译：针对非凸光滑目标的差分隐私优化问题进行研究。此前工作中，以差分隐私梯度下降（DP-GD）为代表的方法在平方全梯度范数下达到的最优效用界为$\widetilde O(\sqrt{d}/(n\varepsilon_\mathrm{DP}))$，其中$n$为样本量，$d$为问题维度，$\varepsilon_\mathrm{DP}$为差分隐私参数。为改进现有最优效用界，我们提出新型差分隐私优化框架——\emph{DIFF2（基于梯度差异的差分隐私优化）}，该框架通过传递\emph{梯度差异}而非梯度本身，构建方差可能极小的差分隐私全局梯度估计器。研究表明，采用梯度下降子程序的DIFF2可实现$\widetilde O(d^{2/3}/(n\varepsilon_\mathrm{DP})^{4/3})$的效用，在样本量$n$的依赖关系上显著优于先前方法。据我们所知，这是首个在非凸目标函数上改进标准效用界$\widetilde O(\sqrt{d}/(n\varepsilon_\mathrm{DP}))$的基础性成果。此外，我们为DIFF2融合了计算与通信效率更高的子程序，并给出其理论分析。数值实验验证了DIFF2框架的优越性。