Zeroth-order (ZO) method has been shown to be a powerful method for solving the optimization problem where explicit expression of the gradients is difficult or infeasible to obtain. Recently, due to the practical value of the constrained problems, a lot of ZO Frank-Wolfe or projected ZO methods have been proposed. However, in many applications, we may have a very large number of nonconvex white/black-box constraints, which makes the existing zeroth-order methods extremely inefficient (or even not working) since they need to inquire function value of all the constraints and project the solution to the complicated feasible set. In this paper, to solve the nonconvex problem with a large number of white/black-box constraints, we proposed a doubly stochastic zeroth-order gradient method (DSZOG) with momentum method and adaptive step size. Theoretically, we prove DSZOG can converge to the $\epsilon$-stationary point of the constrained problem. Experimental results in two applications demonstrate the superiority of our method in terms of training time and accuracy compared with other ZO methods for the constrained problem.

翻译：零阶（ZO）方法已被证明是解决梯度显式表达式难以获取或不可行的优化问题的有力工具。近年来，由于约束问题的实际价值，大量ZO Frank-Wolfe或投影ZO方法被提出。然而，在许多应用中，我们可能面临数量庞大的非凸白盒/黑盒约束，这使得现有零阶方法效率极低（甚至无法工作），因为它们需要查询所有约束的函数值并将解投影到复杂的可行集上。本文针对具有大量白盒/黑盒约束的非凸优化问题，提出了一种结合动量方法与自适应步长的双重随机零阶梯度方法（DSZOG）。理论上，我们证明DSZOG能够收敛到约束问题的$\epsilon$-稳定点。在两个应用场景中的实验结果表明，相较于其他求解约束问题的ZO方法，我们的方法在训练时间与精度方面均展现出优越性。