Recently, there has been significant progress in understanding the convergence and generalization properties of gradient-based methods for training overparameterized learning models. However, many aspects including the role of small random initialization and how the various parameters of the model are coupled during gradient-based updates to facilitate good generalization remain largely mysterious. A series of recent papers have begun to study this role for non-convex formulations of symmetric Positive Semi-Definite (PSD) matrix sensing problems which involve reconstructing a low-rank PSD matrix from a few linear measurements. The underlying symmetry/PSDness is crucial to existing convergence and generalization guarantees for this problem. In this paper, we study a general overparameterized low-rank matrix sensing problem where one wishes to reconstruct an asymmetric rectangular low-rank matrix from a few linear measurements. We prove that an overparameterized model trained via factorized gradient descent converges to the low-rank matrix generating the measurements. We show that in this setting, factorized gradient descent enjoys two implicit properties: (1) coupling of the trajectory of gradient descent where the factors are coupled in various ways throughout the gradient update trajectory and (2) an algorithmic regularization property where the iterates show a propensity towards low-rank models despite the overparameterized nature of the factorized model. These two implicit properties in turn allow us to show that the gradient descent trajectory from small random initialization moves towards solutions that are both globally optimal and generalize well.

翻译：近来，在理解基于梯度的方法训练过参数化学习模型的收敛性与泛化性质方面取得了显著进展。然而，诸多方面仍很大程度上未解，例如小随机初始化的作用，以及梯度更新过程中模型各参数如何耦合以促进良好泛化。近期一系列论文开始研究对称半正定矩阵感知问题的非凸形式中的这一作用，该问题涉及从少量线性测量中重建低秩半正定矩阵。对于该问题，现有的收敛性与泛化保证依赖于潜在的对称性/半正定性。本文研究一个一般的过参数化低秩矩阵感知问题，即希望从少量线性测量中重建一个非对称矩形低秩矩阵。我们证明，通过因子化梯度下降训练的过参数化模型收敛到生成测量值的低秩矩阵。我们表明，在此设置下，因子化梯度下降具有两个隐式特性：（1）梯度下降轨迹的耦合，其中因子在整个梯度更新轨迹中以多种方式相互耦合；（2）算法正则化特性，尽管因子化模型具有过参数化性质，但迭代显示出倾向于低秩模型。这两个隐式特性进而使我们能够证明，从小随机初始化开始的梯度下降轨迹将向全局最优且泛化良好的解移动。