Stochastic optimization methods encounter new challenges in the realm of streaming, characterized by a continuous flow of large, high-dimensional data. While first-order methods, like stochastic gradient descent, are the natural choice, they often struggle with ill-conditioned problems. In contrast, second-order methods, such as Newton's methods, offer a potential solution, but their computational demands render them impractical. This paper introduces adaptive stochastic optimization methods that bridge the gap between addressing ill-conditioned problems while functioning in a streaming context. Notably, we present an adaptive inversion-free Newton's method with a computational complexity matching that of first-order methods, $\mathcal{O}(dN)$, where $d$ represents the number of dimensions/features, and $N$ the number of data. Theoretical analysis confirms their asymptotic efficiency, and empirical evidence demonstrates their effectiveness, especially in scenarios involving complex covariance structures and challenging initializations. In particular, our adaptive Newton's methods outperform existing methods, while maintaining favorable computational efficiency.

翻译：随机优化方法在流式数据处理中面临新挑战，此类数据具有持续涌入、大规模及高维度的特征。虽然随机梯度下降等一阶方法是自然选择，但其在处理病态问题时往往表现不佳。相比之下，牛顿法等二阶方法虽能提供潜在解决方案，却因计算开销过大而难以实用。本文提出能够弥合上述差距的自适应随机优化方法，既具备处理病态问题的能力，又适用于流式场景。特别地，我们提出了一种无需求逆的自适应牛顿法，其计算复杂度与一阶方法相当，为$\mathcal{O}(dN)$，其中$d$表示维度/特征数，$N$表示数据量。理论分析证实了其渐近有效性，实证研究则展示了该方法的优越性能，尤其在处理复杂协方差结构和困难初始化设置时。相较于现有方法，我们的自适应牛顿法在保持计算效率优势的同时，显著提升了解算表现。