We consider sketching algorithms which first compress data by multiplication with a random sketch matrix, and then apply the sketch to quickly solve an optimization problem, e.g., low-rank approximation and regression. In the learning-based sketching paradigm proposed by~\cite{indyk2019learning}, the sketch matrix is found by choosing a random sparse matrix, e.g., CountSketch, and then the values of its non-zero entries are updated by running gradient descent on a training data set. Despite the growing body of work on this paradigm, a noticeable omission is that the locations of the non-zero entries of previous algorithms were fixed, and only their values were learned. In this work, we propose the first learning-based algorithms that also optimize the locations of the non-zero entries. Our first proposed algorithm is based on a greedy algorithm. However, one drawback of the greedy algorithm is its slower training time. We fix this issue and propose approaches for learning a sketching matrix for both low-rank approximation and Hessian approximation for second order optimization. The latter is helpful for a range of constrained optimization problems, such as LASSO and matrix estimation with a nuclear norm constraint. Both approaches achieve good accuracy with a fast running time. Moreover, our experiments suggest that our algorithm can still reduce the error significantly even if we only have a very limited number of training matrices.

翻译：我们考虑素描算法，该算法首先通过乘以随机素描矩阵来压缩数据，然后应用素描快速解决优化问题，例如低秩逼近和回归。在由\cite{indyk2019learning}提出的基于学习的素描范式中，通过选择随机稀疏矩阵（例如CountSketch）来找到素描矩阵，然后通过梯度下降在训练数据集上更新其非零条目的值。尽管关于这一范式的研究日益增多，但一个明显的缺失是，以往算法的非零条目位置是固定的，仅学习其值。在这项工作中，我们首次提出了基于学习的算法，该算法也优化了非零条目的位置。我们提出的第一个算法基于贪心算法。然而，贪心算法的一个缺点是其较慢的训练时间。我们解决了这一问题，并提出了学习素描矩阵的方法，分别用于低秩逼近和二阶优化的Hessian逼近。后者有助于一系列约束优化问题，例如LASSO和具有核范数约束的矩阵估计。这两种方法在快速运行时间内实现了良好的精度。此外，我们的实验表明，即使只有非常有限的训练矩阵，我们的算法仍能显著降低误差。