We consider sketching algorithms which first compress data by multiplication with a random sketch matrix, and then apply the sketch to quickly solve an optimization problem, e.g., low-rank approximation and regression. In the learning-based sketching paradigm proposed by~\cite{indyk2019learning}, the sketch matrix is found by choosing a random sparse matrix, e.g., CountSketch, and then the values of its non-zero entries are updated by running gradient descent on a training data set. Despite the growing body of work on this paradigm, a noticeable omission is that the locations of the non-zero entries of previous algorithms were fixed, and only their values were learned. In this work, we propose the first learning-based algorithms that also optimize the locations of the non-zero entries. Our first proposed algorithm is based on a greedy algorithm. However, one drawback of the greedy algorithm is its slower training time. We fix this issue and propose approaches for learning a sketching matrix for both low-rank approximation and Hessian approximation for second order optimization. The latter is helpful for a range of constrained optimization problems, such as LASSO and matrix estimation with a nuclear norm constraint. Both approaches achieve good accuracy with a fast running time. Moreover, our experiments suggest that our algorithm can still reduce the error significantly even if we only have a very limited number of training matrices.

翻译：我们考虑一类草图算法，该类算法首先通过随机草图矩阵相乘压缩数据，然后利用压缩结果快速求解优化问题（例如低秩近似和回归）。在~\cite{indyk2019learning}提出的基于学习的草图范式框架中，通过选择随机稀疏矩阵（如CountSketch）构建草图矩阵，随后在训练数据集上运行梯度下降来更新其非零元素的数值。尽管该范式的研究日益增多，但一个显著空白是：现有算法的非零元素位置固定不变，仅对其数值进行学习。本文首次提出同时优化非零元素位置的基于学习的算法。我们第一个算法基于贪心策略。然而该算法存在训练时间较慢的缺陷。为解决此问题，我们提出两种方法分别学习用于低秩近似和二阶优化海森矩阵近似的草图矩阵。后者对LASSO、核范数约束矩阵估计等约束优化问题具有重要价值。两种方法在保持快速运行时间的同时均能实现高精度。此外，实验表明，即使仅有极少量训练矩阵，我们的算法仍能显著降低误差。