Stochastic optimization methods encounter new challenges in the realm of streaming, characterized by a continuous flow of large, high-dimensional data. While first-order methods, like stochastic gradient descent, are the natural choice, they often struggle with ill-conditioned problems. In contrast, second-order methods, such as Newton's methods, offer a potential solution, but their computational demands render them impractical. This paper introduces adaptive stochastic optimization methods that bridge the gap between addressing ill-conditioned problems while functioning in a streaming context. Notably, we present an adaptive inversion-free Newton's method with a computational complexity matching that of first-order methods, $\mathcal{O}(dN)$, where $d$ represents the number of dimensions/features, and $N$ the number of data. Theoretical analysis confirms their asymptotic efficiency, and empirical evidence demonstrates their effectiveness, especially in scenarios involving complex covariance structures and challenging initializations. In particular, our adaptive Newton's methods outperform existing methods, while maintaining favorable computational efficiency.

翻译：随机优化方法在流式数据处理领域面临新的挑战，该领域的特点是持续处理大规模高维数据流。虽然随机梯度下降等一阶方法是自然选择，但它们在病态问题上往往表现不佳。相比之下，牛顿法等二阶方法提供了潜在解决方案，但其计算需求使其难以实际应用。本文提出了一种自适应随机优化方法，在保持流式处理能力的同时，有效解决了病态问题。值得注意的是，我们提出了一种自适应免逆牛顿法，其计算复杂度与一阶方法相当，为$\mathcal{O}(dN)$，其中$d$表示维度/特征数量，$N$表示数据量。理论分析验证了其渐近效率，实证结果证明了其有效性，特别是在涉及复杂协方差结构和困难初始化的场景中。具体而言，我们的自适应牛顿法在保持计算效率优势的同时，性能优于现有方法。