Randomized algorithms are important for solving large-scale optimization problems. In this paper, we propose a fast sketching algorithm for least square problems regularized by convex or nonconvex regularization functions, Sketching for Regularized Optimization (SRO). Our SRO algorithm first generates a sketch of the original data matrix, then solves the sketched problem. Different from existing randomized algorithms, our algorithm handles general Frechet subdifferentiable regularization functions in an unified framework. We present general theoretical result for the approximation error between the optimization results of the original problem and the sketched problem for regularized least square problems which can be convex or nonconvex. For arbitrary convex regularizer, relative-error bound is proved for the approximation error. Importantly, minimax rates for sparse signal estimation by solving the sketched sparse convex or nonconvex learning problems are also obtained using our general theoretical result under mild conditions. To the best of our knowledge, our results are among the first to demonstrate minimax rates for convex or nonconvex sparse learning problem by sketching under a unified theoretical framework. We further propose an iterative sketching algorithm which reduces the approximation error exponentially by iteratively invoking the sketching algorithm. Experimental results demonstrate the effectiveness of the proposed SRO and Iterative SRO algorithms.

翻译：随机算法在解决大规模优化问题中具有重要意义。本文针对凸或非凸正则化函数约束的最小二乘问题，提出了一种快速草图算法——正则化优化草图算法（SRO）。SRO算法首先生成原始数据矩阵的草图，随后求解草图化问题。与现有随机算法不同，该算法在统一框架下处理一般形式的Fréchet次可微正则化函数。针对凸或非凸的正则化最小二乘问题，我们给出了原始问题与草图化问题优化结果之间近似误差的通用理论结论。对于任意凸正则化器，证明了近似误差的相对误差界。重要的是，基于本通用理论结果，我们在温和条件下进一步获得了通过求解草图化稀疏凸/非凸学习问题实现稀疏信号估计的极小极大速率。据我们所知，本研究是首个在统一理论框架下通过草图算法证明凸/非凸稀疏学习问题极小极大速率的成果。我们进一步提出迭代草图算法，通过重复调用草图算法将近似误差呈指数级降低。实验结果表明，所提出的SRO算法与迭代SRO算法具有显著有效性。