Differential private optimization for nonconvex smooth objective is considered. In the previous work, the best known utility bound is $\widetilde O(\sqrt{d}/(n\varepsilon_\mathrm{DP}))$ in terms of the squared full gradient norm, which is achieved by Differential Private Gradient Descent (DP-GD) as an instance, where $n$ is the sample size, $d$ is the problem dimensionality and $\varepsilon_\mathrm{DP}$ is the differential privacy parameter. To improve the best known utility bound, we propose a new differential private optimization framework called \emph{DIFF2 (DIFFerential private optimization via gradient DIFFerences)} that constructs a differential private global gradient estimator with possibly quite small variance based on communicated \emph{gradient differences} rather than gradients themselves. It is shown that DIFF2 with a gradient descent subroutine achieves the utility of $\widetilde O(d^{2/3}/(n\varepsilon_\mathrm{DP})^{4/3})$, which can be significantly better than the previous one in terms of the dependence on the sample size $n$. To the best of our knowledge, this is the first fundamental result to improve the standard utility $\widetilde O(\sqrt{d}/(n\varepsilon_\mathrm{DP}))$ for nonconvex objectives. Additionally, a more computational and communication efficient subroutine is combined with DIFF2 and its theoretical analysis is also given. Numerical experiments are conducted to validate the superiority of DIFF2 framework.

翻译：研究考虑非凸光滑目标的差分隐私优化问题。此前工作中，基于平方全梯度范数的最佳已知效用界为$\widetilde O(\sqrt{d}/(n\varepsilon_\mathrm{DP}))$，该界由差分隐私梯度下降（DP-GD）实现，其中$n$为样本量，$d$为问题维度，$\varepsilon_\mathrm{DP}$为差分隐私参数。为提升该最佳已知效用界，我们提出一种新的差分隐私优化框架——\emph{DIFF2（基于梯度差异的差分隐私优化）}，该框架通过通信\emph{梯度差异}而非梯度本身，构建具有极小方差的差分隐私全局梯度估计器。研究表明，采用梯度下降子程序的DIFF2可实现$\widetilde O(d^{2/3}/(n\varepsilon_\mathrm{DP})^{4/3})$的效用，在样本量$n$的依赖关系上显著优于此前结果。据我们所知，这是首个从根本上提升非凸目标函数标准效用界$\widetilde O(\sqrt{d}/(n\varepsilon_\mathrm{DP}))$的成果。此外，本文还将一种计算与通信效率更高的子程序与DIFF2相结合，并给出其理论分析。通过数值实验验证了DIFF2框架的优越性。