While overparameterization in machine learning models offers great benefits in terms of optimization and generalization, it also leads to increased computational requirements as model sizes grow. In this work, we show that by leveraging the inherent low-dimensional structures of data and compressible dynamics within the model parameters, we can reap the benefits of overparameterization without the computational burdens. In practice, we demonstrate the effectiveness of this approach for deep low-rank matrix completion as well as fine-tuning language models. Our approach is grounded in theoretical findings for deep overparameterized low-rank matrix recovery, where we show that the learning dynamics of each weight matrix are confined to an invariant low-dimensional subspace. Consequently, we can construct and train compact, highly compressed factorizations possessing the same benefits as their overparameterized counterparts. In the context of deep matrix completion, our technique substantially improves training efficiency while retaining the advantages of overparameterization. For language model fine-tuning, we propose a method called "Deep LoRA", which improves the existing low-rank adaptation (LoRA) technique, leading to reduced overfitting and a simplified hyperparameter setup, while maintaining comparable efficiency. We validate the effectiveness of Deep LoRA on natural language tasks, particularly when fine-tuning with limited data. Our code is available at https://github.com/cjyaras/deep-lora-transformers.

翻译：尽管机器学习模型中的过参数化在优化和泛化方面带来了巨大益处，但随着模型规模的增长，它也导致了计算需求的增加。在本工作中，我们表明，通过利用数据固有的低维结构以及模型参数内的可压缩动力学，我们能够获得过参数化的好处，而无需承担其计算负担。在实践中，我们证明了该方法在深度低秩矩阵补全以及语言模型微调中的有效性。我们的方法基于深度过参数化低秩矩阵恢复的理论发现，其中我们证明了每个权重矩阵的学习动力学都被限制在一个不变的低维子空间内。因此，我们可以构建并训练紧凑、高度压缩的分解形式，这些形式拥有与其过参数化对应物相同的优势。在深度矩阵补全的背景下，我们的技术显著提高了训练效率，同时保留了过参数化的优点。对于语言模型微调，我们提出了一种名为"Deep LoRA"的方法，该方法改进了现有的低秩适应（LoRA）技术，从而减少了过拟合并简化了超参数设置，同时保持了相当的效率。我们在自然语言任务上验证了Deep LoRA的有效性，特别是在使用有限数据进行微调时。我们的代码可在 https://github.com/cjyaras/deep-lora-transformers 获取。