The zeroth-order optimization has been widely used in machine learning applications. However, the theoretical study of the zeroth-order optimization focus on the algorithms which approximate (first-order) gradients using (zeroth-order) function value difference at a random direction. The theory of algorithms which approximate the gradient and Hessian information by zeroth-order queries is much less studied. In this paper, we focus on the theory of zeroth-order optimization which utilizes both the first-order and second-order information approximated by the zeroth-order queries. We first propose a novel reparameterized objective function with parameters $(\mu, \Sigma)$. This reparameterized objective function achieves its optimum at the minimizer and the Hessian inverse of the original objective function respectively, but with small perturbations. Accordingly, we propose a new algorithm to minimize our proposed reparameterized objective, which we call \texttt{MiNES} (mirror descent natural evolution strategy). We show that the estimated covariance matrix of \texttt{MiNES} converges to the inverse of Hessian matrix of the objective function with a convergence rate $\widetilde{\mathcal{O}}(1/k)$, where $k$ is the iteration number and $\widetilde{\mathcal{O}}(\cdot)$ hides the constant and $\log$ terms. We also provide the explicit convergence rate of \texttt{MiNES} and how the covariance matrix promotes the convergence rate.

翻译：零阶优化已在机器学习应用中得到广泛使用。然而，零阶优化的理论研究主要聚焦于通过随机方向上的（零阶）函数值差来近似（一阶）梯度的算法。针对通过零阶查询同时近似梯度与海森信息的算法，其理论研究尚不充分。本文重点研究利用零阶查询近似的一阶与二阶信息的零阶优化理论。首先，我们提出一种带有参数$(\mu, \Sigma)$的新型重参数化目标函数。该重参数化目标函数分别在原目标函数的最小值点与海森逆矩阵处达到最优（仅存在微小扰动）。据此，我们提出一种最小化该重参数化目标函数的新算法，称之为\texttt{MiNES}（镜像下降自然进化策略）。我们证明\texttt{MiNES}估计的协方差矩阵以收敛速率$\widetilde{\mathcal{O}}(1/k)$收敛至目标函数海森矩阵的逆矩阵，其中$k$为迭代次数，$\widetilde{\mathcal{O}}(\cdot)$隐藏了常数项与$\log$项。我们还给出了\texttt{MiNES}的显式收敛速率，并阐释了协方差矩阵如何促进收敛速率的提升。