In this work, we develop first-order (Hessian-free) and zero-order (derivative-free) implementations of the Cubically regularized Newton method for solving general non-convex optimization problems. For that, we employ finite difference approximations of the derivatives. We use a special adaptive search procedure in our algorithms, which simultaneously fits both the regularization constant and the parameters of the finite difference approximations. It makes our schemes free from the need to know the actual Lipschitz constants. Additionally, we equip our algorithms with the lazy Hessian update that reuse a previously computed Hessian approximation matrix for several iterations. Specifically, we prove the global complexity bound of $\mathcal{O}( n^{1/2} \epsilon^{-3/2})$ function and gradient evaluations for our new Hessian-free method, and a bound of $\mathcal{O}( n^{3/2} \epsilon^{-3/2} )$ function evaluations for the derivative-free method, where $n$ is the dimension of the problem and $\epsilon$ is the desired accuracy for the gradient norm. These complexity bounds significantly improve the previously known ones in terms of the joint dependence on $n$ and $\epsilon$, for the first-order and zeroth-order non-convex optimization.

翻译：本文针对一般非凸优化问题，开发了三次正则化牛顿方法的一阶（无Hessian）和零阶（无导数）实现。为此，我们采用导数的有限差分近似。算法中引入了一种自适应搜索策略，该策略同时拟合正则化常数与有限差分参数，从而无需获知实际Lipschitz常数。此外，我们为算法配备了懒散Hessian更新机制，允许在多次迭代中复用先前计算的Hessian近似矩阵。具体而言，我们证明了新无Hessian方法需进行$\mathcal{O}( n^{1/2} \epsilon^{-3/2})$次函数与梯度评估的全局复杂度界，以及无导数方法需进行$\mathcal{O}( n^{3/2} \epsilon^{-3/2} )$次函数评估的复杂度界，其中$n$为问题维度，$\epsilon$为梯度范数的期望精度。在一阶与零阶非凸优化领域，这些复杂度界在$n$与$\epsilon$的联合依赖关系上显著优于此前已知结果。