The paper discusses derivative-free optimization (DFO), which involves minimizing a function without access to gradients or directional derivatives, only function evaluations. Classical DFO methods, which mimic gradient-based methods, such as Nelder-Mead and direct search have limited scalability for high-dimensional problems. Zeroth-order methods have been gaining popularity due to the demands of large-scale machine learning applications, and the paper focuses on the selection of the step size $\alpha_k$ in these methods. The proposed approach, called Curvature-Aware Random Search (CARS), uses first- and second-order finite difference approximations to compute a candidate $\alpha_{+}$. We prove that for strongly convex objective functions, CARS converges linearly provided that the search direction is drawn from a distribution satisfying very mild conditions. We also present a Cubic Regularized variant of CARS, named CARS-CR, which converges in a rate of $\mathcal{O}(k^{-1})$ without the assumption of strong convexity. Numerical experiments show that CARS and CARS-CR match or exceed the state-of-the-arts on benchmark problem sets.

翻译：本文讨论无导数优化(DFO)问题，即在无法获取梯度或方向导数、仅能利用函数评估值的情况下最小化目标函数。经典DFO方法（如Nelder-Mead和直接搜索法）通过模仿梯度方法进行优化，但在高维问题中可扩展性有限。由于大规模机器学习应用的需求，零阶方法近年来日益流行，本文重点关注这类方法中步长$\alpha_k$的选取策略。我们提出名为"曲率感知随机搜索(CARS)"的新方法，通过一阶和二阶有限差分逼近计算候选步长$\alpha_{+}$。理论证明，在强凸目标函数下，只要搜索方向服从满足极弱条件的分布，CARS即可实现线性收敛。此外，我们提出CARS的三次正则化变体CARS-CR，该变体在无需强凸假设条件下达到$\mathcal{O}(k^{-1})$收敛速率。数值实验表明，CARS与CARS-CR在基准测试集上的表现可与当前最优方法媲美甚至更优。