We provide a simple and flexible framework for designing differentially private algorithms to find approximate stationary points of non-convex loss functions. Our framework is based on using a private approximate risk minimizer to "warm start" another private algorithm for finding stationary points. We use this framework to obtain improved, and sometimes optimal, rates for several classes of non-convex loss functions. First, we obtain improved rates for finding stationary points of smooth non-convex empirical loss functions. Second, we specialize to quasar-convex functions, which generalize star-convex functions and arise in learning dynamical systems and training some neural nets. We achieve the optimal rate for this class. Third, we give an optimal algorithm for finding stationary points of functions satisfying the Kurdyka-Lojasiewicz (KL) condition. For example, over-parameterized neural networks often satisfy this condition. Fourth, we provide new state-of-the-art rates for stationary points of non-convex population loss functions. Fifth, we obtain improved rates for non-convex generalized linear models. A modification of our algorithm achieves nearly the same rates for second-order stationary points of functions with Lipschitz Hessian, improving over the previous state-of-the-art for each of the above problems.

翻译：我们提供了一个简单而灵活的框架，用于设计差分隐私算法以寻找非凸损失函数的近似驻点。该框架基于使用私有近似风险最小化器来“热启动”另一个用于寻找驻点的私有算法。利用此框架，我们针对几类非凸损失函数获得了改进甚至最优的速率。首先，我们获得了寻找光滑非凸经验损失函数驻点的改进速率。其次，我们专门研究了拟凸函数——它推广了星凸函数，并在学习动力系统和训练某些神经网络中出现。我们为此类函数达到了最优速率。第三，我们给出了一个满足Kurdyka-Lojasiewicz（KL）条件的函数寻找驻点的最优算法。例如，过参数化神经网络通常满足该条件。第四，我们为非凸总体损失函数驻点提供了新的最优速率。第五，我们为非凸广义线性模型获得了改进速率。我们算法的变体在寻找具有Lipschitz Hessian函数的二阶驻点时几乎达到相同速率，从而改进了上述每个问题之前的最优结果。