We propose $\textsf{ScaledGD($\lambda$)}$, a preconditioned gradient descent method to tackle the low-rank matrix sensing problem when the true rank is unknown, and when the matrix is possibly ill-conditioned. Using overparametrized factor representations, $\textsf{ScaledGD($\lambda$)}$ starts from a small random initialization, and proceeds by gradient descent with a specific form of damped preconditioning to combat bad curvatures induced by overparameterization and ill-conditioning. At the expense of light computational overhead incurred by preconditioners, $\textsf{ScaledGD($\lambda$)}$ is remarkably robust to ill-conditioning compared to vanilla gradient descent ($\textsf{GD}$) even with overprameterization. Specifically, we show that, under the Gaussian design, $\textsf{ScaledGD($\lambda$)}$ converges to the true low-rank matrix at a constant linear rate after a small number of iterations that scales only logarithmically with respect to the condition number and the problem dimension. This significantly improves over the convergence rate of vanilla $\textsf{GD}$ which suffers from a polynomial dependency on the condition number. Our work provides evidence on the power of preconditioning in accelerating the convergence without hurting generalization in overparameterized learning.

翻译：我们提出$\textsf{ScaledGD($\lambda$)}$，一种预条件梯度下降方法，用于解决真实秩未知且矩阵可能病态的低秩矩阵感知问题。利用过参数化因子表示，$\textsf{ScaledGD($\lambda$)}$从小的随机初始化开始，并通过一种特定形式的阻尼预条件梯度下降进行迭代，以对抗过参数化和病态性引起的不良曲率。尽管预条件器引入了轻微的计算开销，但与普通梯度下降（$\textsf{GD}$）相比，即使采用过参数化，$\textsf{ScaledGD($\lambda$)}$对病态性也表现出显著的鲁棒性。具体而言，我们证明，在高斯设计下，$\textsf{ScaledGD($\lambda$)}$在经过少量迭代后以恒定的线性速率收敛到真实低秩矩阵，该迭代次数仅关于条件数和问题维度呈对数增长。这显著优于普通$\textsf{GD}$的收敛速率，后者在条件数上依赖于多项式关系。我们的工作为预条件在过参数化学习中加速收敛而不损害泛化能力提供了证据。