Stochastic versions of proximal methods have gained much attention in statistics and machine learning. These algorithms tend to admit simple, scalable forms, and enjoy numerical stability via implicit updates. In this work, we propose and analyze a stochastic version of the recently proposed proximal distance algorithm, a class of iterative optimization methods that recover a desired constrained estimation problem as a penalty parameter $\rho \rightarrow \infty$. By uncovering connections to related stochastic proximal methods and interpreting the penalty parameter as the learning rate, we justify heuristics used in practical manifestations of the proximal distance method, establishing their convergence guarantees for the first time. Moreover, we extend recent theoretical devices to establish finite error bounds and a complete characterization of convergence rates regimes. We validate our analysis via a thorough empirical study, also showing that unsurprisingly, the proposed method outpaces batch versions on popular learning tasks.

翻译：随机版本的近端方法在统计学和机器学习领域引起了广泛关注。这类算法通常具有简单、可扩展的形式，并通过隐式更新享有数值稳定性。在本文中，我们提出并分析了一种新近提出的近端距离算法的随机版本——这是一类迭代优化方法，通过惩罚参数 $\rho \rightarrow \infty$ 恢复所需的约束估计问题。通过揭示与相关随机近端方法的联系，并将惩罚参数解释为学习率，我们为近端距离方法实际应用中所使用的启发式策略提供了理论依据，首次建立了其收敛性保证。此外，我们扩展了最新理论工具，建立了有限误差界并对收敛速率区域进行了完整刻画。通过详尽的实证研究验证了我们的分析，同时表明，毫不意外地，所提出的方法在常见学习任务上优于批处理版本。