We consider the problem of minimizing a non-convex objective while preserving the privacy of the examples in the training data. Building upon the previous variance-reduced algorithm SpiderBoost, we introduce a new framework that utilizes two different kinds of gradient oracles. The first kind of oracles can estimate the gradient of one point, and the second kind of oracles, less precise and more cost-effective, can estimate the gradient difference between two points. SpiderBoost uses the first kind periodically, once every few steps, while our framework proposes using the first oracle whenever the total drift has become large and relies on the second oracle otherwise. This new framework ensures the gradient estimations remain accurate all the time, resulting in improved rates for finding second-order stationary points. Moreover, we address a more challenging task of finding the global minima of a non-convex objective using the exponential mechanism. Our findings indicate that the regularized exponential mechanism can closely match previous empirical and population risk bounds, without requiring smoothness assumptions for algorithms with polynomial running time. Furthermore, by disregarding running time considerations, we show that the exponential mechanism can achieve a good population risk bound and provide a nearly matching lower bound.

翻译：本文研究在保护训练数据样本隐私的前提下最小化非凸目标函数的问题。基于先前的方差缩减算法SpiderBoost，我们引入了一种利用两种不同梯度预言机的新框架。第一类预言机可估计单点梯度，第二类预言机虽精度较低但成本更优，可估计两点间的梯度差异。SpiderBoost以固定间隔周期性使用第一类预言机，而我们的框架提出在总漂移量较大时启用第一类预言机，其余情况则依赖第二类预言机。该新框架确保梯度估计始终保持高精度，从而改进了寻找二阶驻点的收敛速率。此外，我们通过指数机制解决了寻找非凸目标全局最小值这一更具挑战性的任务。研究结果表明，正则化指数机制能够在无需光滑性假设的条件下，以多项式运行时间算法紧密匹配先前的经验风险与总体风险边界。进一步地，忽略运行时间约束时，指数机制可实现优良的总体风险边界，并给出近乎匹配的下界。