Stochastic approximation (SA) is a classical algorithm that has had since the early days a huge impact on signal processing, and nowadays on machine learning, due to the necessity to deal with a large amount of data observed with uncertainties. An exemplar special case of SA pertains to the popular stochastic (sub)gradient algorithm which is the working horse behind many important applications. A lesser-known fact is that the SA scheme also extends to non-stochastic-gradient algorithms such as compressed stochastic gradient, stochastic expectation-maximization, and a number of reinforcement learning algorithms. The aim of this article is to overview and introduce the non-stochastic-gradient perspectives of SA to the signal processing and machine learning audiences through presenting a design guideline of SA algorithms backed by theories. Our central theme is to propose a general framework that unifies existing theories of SA, including its non-asymptotic and asymptotic convergence results, and demonstrate their applications on popular non-stochastic-gradient algorithms. We build our analysis framework based on classes of Lyapunov functions that satisfy a variety of mild conditions. We draw connections between non-stochastic-gradient algorithms and scenarios when the Lyapunov function is smooth, convex, or strongly convex. Using the said framework, we illustrate the convergence properties of the non-stochastic-gradient algorithms using concrete examples. Extensions to the emerging variance reduction techniques for improved sample complexity will also be discussed.

翻译：随机逼近（SA）是一种经典算法，自诞生之初便对信号处理领域产生深远影响，如今更因处理含不确定性海量数据的现实需求，在机器学习领域发挥关键作用。其特例——流行的随机（次）梯度算法——已成为众多重要应用的核心引擎。鲜为人知的是，SA框架还能拓展至非随机梯度算法，包括压缩随机梯度、随机期望最大化以及多种强化学习算法。本文旨在向信号处理与机器学习领域的研究者系统介绍SA的非随机梯度视角，通过呈现基于理论支撑的SA算法设计准则，提出一个统一现有SA理论（包括渐近与非渐近收敛性分析）的通用框架，并阐释其在典型非随机梯度算法中的应用。我们基于满足系列温和条件的李雅普诺夫函数类构建分析框架，揭示非随机梯度算法在李雅普诺夫函数满足光滑性、凸性或强凸性等不同条件下的算法特性，通过具体实例演示其收敛性质。此外，还将探讨提升样本复杂度的新兴方差缩减技术拓展方案。