Given the ubiquity of streaming data, online algorithms have been widely used for parameter estimation, with second-order methods particularly standing out for their efficiency and robustness. In this paper, we study an online sketched Newton method that leverages a randomized sketching technique to perform an approximate Newton step in each iteration, thereby eliminating the computational bottleneck of second-order methods. While existing studies have established the asymptotic normality of sketched Newton methods, a consistent estimator of the limiting covariance matrix remains an open problem. We propose a fully online covariance matrix estimator that is constructed entirely from the Newton iterates and requires no matrix factorization. Compared to covariance estimators for first-order online methods, our estimator for second-order methods is batch-free. We establish the consistency and convergence rate of our estimator, and coupled with asymptotic normality results, we can then perform online statistical inference for the model parameters based on sketched Newton methods. We also discuss the extension of our estimator to constrained problems, and demonstrate its superior performance on regression problems as well as benchmark problems in the CUTEst set.

翻译：鉴于流式数据的普遍性，在线算法已被广泛用于参数估计，其中二阶方法因其高效性和鲁棒性而尤为突出。本文研究了一种在线素描牛顿法，该方法利用随机素描技术在每次迭代中执行近似牛顿步，从而消除了二阶方法的计算瓶颈。尽管现有研究已建立了素描牛顿法的渐近正态性，但其极限协方差矩阵的一致估计仍是一个开放问题。我们提出了一种完全在线的协方差矩阵估计器，该估计器完全基于牛顿迭代构建，且无需矩阵分解。与一阶在线方法的协方差估计器相比，我们的二阶方法估计器无需批处理。我们建立了该估计器的一致性和收敛速率，并结合渐近正态性结果，能够基于素描牛顿法对模型参数进行在线统计推断。我们还讨论了该估计器在约束问题上的扩展，并在回归问题及CUTEst基准测试集上展示了其优越性能。