Stochastic optimisation algorithms are the de facto standard for machine learning with large amounts of data. Handling only a subset of available data in each optimisation step dramatically reduces the per-iteration computational costs, while still ensuring significant progress towards the solution. Driven by the need to solve large-scale optimisation problems as efficiently as possible, the last decade has witnessed an explosion of research in this area. Leveraging the parallels between machine learning and inverse problems has allowed harnessing the power of this research wave for solving inverse problems. In this survey, we provide a comprehensive account of the state-of-the-art in stochastic optimisation from the viewpoint of variational regularisation for inverse problems where the solution is modelled as minimising an objective function. We present algorithms with diverse modalities of problem randomisation and discuss the roles of variance reduction, acceleration, higher-order methods, and other algorithmic modifications, and compare theoretical results with practical behaviour. We focus on the potential and the challenges for stochastic optimisation that are unique to variational regularisation for inverse imaging problems and are not commonly encountered in machine learning. We conclude the survey with illustrative examples from imaging on linear inverse problems to examine the advantages and disadvantages that this new generation of algorithms bring to the field of inverse problems.

翻译：随机优化算法已成为处理海量数据机器学习的事实标准。在每次优化迭代中仅处理可用数据的一个子集，能显著降低单次迭代的计算成本，同时仍确保向解方向取得实质性进展。在尽可能高效求解大规模优化问题的需求驱动下，过去十年该领域研究呈现爆发式增长。通过利用机器学习与逆问题之间的相似性，这股研究浪潮的成果得以成功应用于逆问题求解。本综述从变分正则化视角（其解建模为目标函数最小化问题），系统阐述随机优化在逆问题领域的前沿进展。我们展示具有不同问题随机化模式的算法，探讨方差缩减、加速技术、高阶方法及其他算法改进的作用，并将理论结果与实际性能进行对比。重点关注随机优化在逆成像问题变分正则化中特有的潜力与挑战——这些是机器学习领域不常遇到的。最后通过线性逆成像问题的示例，剖析新一代算法为逆问题领域带来的优势与局限。