We study stochastic zeroth order gradient and Hessian estimators for real-valued functions in $\mathbb{R}^n$. We show that, via taking finite difference along random orthogonal directions, the variance of the stochastic finite difference estimators can be significantly reduced. In particular, we design estimators for smooth functions such that, if one uses $ \Theta \left( k \right) $ random directions sampled from the Stiefel's manifold $ \text{St} (n,k) $ and finite-difference granularity $\delta$, the variance of the gradient estimator is bounded by $ \mathcal{O} \left( \left( \frac{n}{k} - 1 \right) + \left( \frac{n^2}{k} - n \right) \delta^2 + \frac{ n^2 \delta^4 }{ k } \right) $, and the variance of the Hessian estimator is bounded by $\mathcal{O} \left( \left( \frac{n^2}{k^2} - 1 \right) + \left( \frac{n^4}{k^2} - n^2 \right) \delta^2 + \frac{n^4 \delta^4 }{k^2} \right) $. When $k = n$, the variances become negligibly small. In addition, we provide improved bias bounds for the estimators. The bias of both gradient and Hessian estimators for smooth function $f$ is of order $\mathcal{O} \left( \delta^2 \Gamma \right)$, where $\delta$ is the finite-difference granularity, and $ \Gamma $ depends on high order derivatives of $f$. Our results are evidenced by empirical observations.

翻译：我们研究实值函数在 $\mathbb{R}^n$ 中的随机零阶梯度和海森矩阵估计量。研究表明，通过沿随机正交方向进行有限差分，随机有限差分估计量的方差可显著降低。具体而言，针对光滑函数设计的估计量表明：若从施蒂费尔流形 $\text{St} (n,k)$ 中采样 $\Theta \left( k \right)$ 个随机方向，并采用有限差分粒度 $\delta$，则梯度估计量的方差上界为 $\mathcal{O} \left( \left( \frac{n}{k} - 1 \right) + \left( \frac{n^2}{k} - n \right) \delta^2 + \frac{ n^2 \delta^4 }{ k } \right)$，海森矩阵估计量的方差上界为 $\mathcal{O} \left( \left( \frac{n^2}{k^2} - 1 \right) + \left( \frac{n^4}{k^2} - n^2 \right) \delta^2 + \frac{n^4 \delta^4 }{k^2} \right)$。当 $k = n$ 时，方差可忽略不计。此外，我们改进了估计量的偏差界：对光滑函数 $f$，梯度和海森矩阵估计量的偏差均为 $\mathcal{O} \left( \delta^2 \Gamma \right)$ 量级，其中 $\delta$ 为有限差分粒度，$\Gamma$ 取决于 $f$ 的高阶导数。实证观察验证了上述结论。