Zeroth-order (a.k.a, derivative-free) methods are a class of effective optimization methods for solving complex machine learning problems, where gradients of the objective functions are not available or computationally prohibitive. Recently, although many zeroth-order methods have been developed, these approaches still have two main drawbacks: 1) high function query complexity; 2) not being well suitable for solving the problems with complex penalties and constraints. To address these challenging drawbacks, in this paper, we propose a class of faster zeroth-order stochastic alternating direction method of multipliers (ADMM) methods (ZO-SPIDER-ADMM) to solve the nonconvex finite-sum problems with multiple nonsmooth penalties. Moreover, we prove that the ZO-SPIDER-ADMM methods can achieve a lower function query complexity of $O(nd+dn^{\frac{1}{2}}\epsilon^{-1})$ for finding an $\epsilon$-stationary point, which improves the existing best nonconvex zeroth-order ADMM methods by a factor of $O(d^{\frac{1}{3}}n^{\frac{1}{6}})$, where $n$ and $d$ denote the sample size and data dimension, respectively. At the same time, we propose a class of faster zeroth-order online ADMM methods (ZOO-ADMM+) to solve the nonconvex online problems with multiple nonsmooth penalties. We also prove that the proposed ZOO-ADMM+ methods achieve a lower function query complexity of $O(d\epsilon^{-\frac{3}{2}})$, which improves the existing best result by a factor of $O(\epsilon^{-\frac{1}{2}})$. Extensive experimental results on the structure adversarial attack on black-box deep neural networks demonstrate the efficiency of our new algorithms.

翻译：零阶（即无导数）方法是一类高效的优化方法，用于求解目标函数梯度不可用或计算代价高昂的复杂机器学习问题。尽管近年来已发展出许多零阶方法，但这些方法仍存在两个主要缺陷：1）函数查询复杂度高；2）难以有效处理含复杂惩罚项和约束的问题。为应对这些挑战性缺陷，本文提出了一类更快的零阶随机交替方向乘子法（ZO-SPIDER-ADMM），用于求解含多个非光滑惩罚项的非凸有限和问题。此外，我们证明ZO-SPIDER-ADMM方法在寻找$\epsilon$-稳定点时能达到更低的函数查询复杂度$O(nd+dn^{\frac{1}{2}}\epsilon^{-1})$，相比现有最优非凸零阶ADMM方法提升了$O(d^{\frac{1}{3}}n^{\frac{1}{6}})$倍（其中$n$和$d$分别表示样本量和数据维度）。同时，我们提出一类更快的零阶在线ADMM方法（ZOO-ADMM+）以求解含多个非光滑惩罚项的非凸在线问题，并证明所提ZOO-ADMM+方法能达到$O(d\epsilon^{-\frac{3}{2}})$的更低函数查询复杂度，相比现有最优结果提升了$O(\epsilon^{-\frac{1}{2}})$倍。在黑盒深度神经网络结构对抗攻击上的大量实验结果表明了所提新算法的高效性。