A substantial body of work in machine learning (ML) and randomized numerical linear algebra (RandNLA) has exploited various sorts of random sketching methodologies, including random sampling and random projection, with much of the analysis using Johnson--Lindenstrauss and subspace embedding techniques. Recent studies have identified the issue of inversion bias -- the phenomenon that inverses of random sketches are not unbiased, despite the unbiasedness of the sketches themselves. This bias presents challenges for the use of random sketches in various ML pipelines, such as fast stochastic optimization, scalable statistical estimators, and distributed optimization. In the context of random projection, the inversion bias can be easily corrected for dense Gaussian projections (which are, however, too expensive for many applications). Recent work has shown how the inversion bias can be corrected for sparse sub-gaussian projections. In this paper, we show how the inversion bias can be corrected for random sampling methods, both uniform and non-uniform leverage-based, as well as for structured random projections, including those based on the Hadamard transform. Using these results, we establish problem-independent local convergence rates for sub-sampled Newton methods.

翻译：机器学习（ML）与随机数值线性代数（RandNLA）领域的大量研究利用了多种随机草图方法，包括随机采样与随机投影，其分析多基于Johnson–Lindenstrauss定理与子空间嵌入技术。近期研究揭示了求逆偏差问题——即随机草图的逆矩阵存在偏差，尽管草图本身具有无偏性。该偏差对随机草图在多种ML流程中的应用构成挑战，例如快速随机优化、可扩展统计估计器及分布式优化。在随机投影场景中，稠密高斯投影的求逆偏差易于校正（但此类投影的计算成本对多数应用而言过高）。最新研究表明稀疏次高斯投影的求逆偏差亦可被修正。本文系统阐述了如何校正随机采样方法（包括均匀采样与非均匀杠杆采样）及结构化随机投影（如基于哈达玛变换的投影）的求逆偏差。基于这些结果，我们建立了子采样牛顿方法中与问题无关的局部收敛速率。