Recently, there has been significant progress in understanding the convergence and generalization properties of gradient-based methods for training overparameterized learning models. However, many aspects including the role of small random initialization and how the various parameters of the model are coupled during gradient-based updates to facilitate good generalization remain largely mysterious. A series of recent papers have begun to study this role for non-convex formulations of symmetric Positive Semi-Definite (PSD) matrix sensing problems which involve reconstructing a low-rank PSD matrix from a few linear measurements. The underlying symmetry/PSDness is crucial to existing convergence and generalization guarantees for this problem. In this paper, we study a general overparameterized low-rank matrix sensing problem where one wishes to reconstruct an asymmetric rectangular low-rank matrix from a few linear measurements. We prove that an overparameterized model trained via factorized gradient descent converges to the low-rank matrix generating the measurements. We show that in this setting, factorized gradient descent enjoys two implicit properties: (1) coupling of the trajectory of gradient descent where the factors are coupled in various ways throughout the gradient update trajectory and (2) an algorithmic regularization property where the iterates show a propensity towards low-rank models despite the overparameterized nature of the factorized model. These two implicit properties in turn allow us to show that the gradient descent trajectory from small random initialization moves towards solutions that are both globally optimal and generalize well.

翻译：近年来，在理解基于梯度的方法训练过参数化学习模型的收敛性与泛化性质方面取得了显著进展。然而，许多方面仍然存在诸多谜团，包括小随机初始化的作用以及梯度更新过程中模型各参数如何相互耦合以促进良好泛化。最近一系列论文开始研究对称半正定矩阵感知问题非凸形式的这一作用——该问题涉及从少量线性测量中重构低秩半正定矩阵。现有针对该问题的收敛性与泛化保证依赖于其对称性/半正定性。在本文中，我们研究一个通用的过参数化低秩矩阵感知问题：目标是从少量线性测量中重构非对称矩形低秩矩阵。我们证明通过因子化梯度下降训练的过参数化模型收敛于生成测量值的低秩矩阵。我们证明在此设定下，因子化梯度下降具有两种隐式性质：(1) 梯度下降轨迹的耦合性——在梯度更新轨迹中因子以多种方式相互耦合；(2) 算法正则化性质——尽管因子化模型具有过参数化特性，迭代过程仍倾向于低秩模型。这两种隐式性质进而使我们能够证明：从小的随机初始化出发的梯度下降轨迹趋向于全局最优且泛化良好的解。