In this paper, we provide a novel analytical perspective on the theoretical understanding of gradient-based learning algorithms by interpreting consensus-based optimization (CBO), a recently proposed multi-particle derivative-free optimization method, as a stochastic relaxation of gradient descent. Remarkably, we observe that through communication of the particles, CBO exhibits a stochastic gradient descent (SGD)-like behavior despite solely relying on evaluations of the objective function. The fundamental value of such link between CBO and SGD lies in the fact that CBO is provably globally convergent to global minimizers for ample classes of nonsmooth and nonconvex objective functions. Hence, on the one side, we offer a novel explanation for the success of stochastic relaxations of gradient descent by furnishing useful and precise insights that explain how problem-tailored stochastic perturbations of gradient descent (like the ones induced by CBO) overcome energy barriers and reach deep levels of nonconvex functions. On the other side, and contrary to the conventional wisdom for which derivative-free methods ought to be inefficient or not to possess generalization abilities, our results unveil an intrinsic gradient descent nature of heuristics. Instructive numerical illustrations support the provided theoretical insights.

翻译：本文通过将共识优化——一种近期提出的无导数多粒子优化方法——诠释为梯度下降的随机松弛，为基于梯度的学习算法的理论理解提供了新颖的分析视角。值得注意的是，我们观察到：尽管仅依赖目标函数值的评估，CBO通过粒子间的通信展现出类随机梯度下降的行为。CBO与SGD之间这种联系的根本价值在于，CBO可被严格证明对广泛类型的非光滑非凸目标函数具有全局收敛到全局极小值的性质。因此，一方面我们通过提供精确而有用的见解，解释了针对问题定制的梯度下降随机扰动（如CBO所诱导的扰动）如何跨越能量壁垒并抵达非凸函数的深层区域，从而为梯度下降随机松弛方法的成功提供了新颖解释。另一方面，与传统认为无导数方法必然低效或缺乏泛化能力的观点相反，我们的研究结果揭示了启发式方法内在的梯度下降本质。具有启发性的数值算例为所提出的理论见解提供了支撑。