Despite large neural networks demonstrating remarkable abilities to complete different tasks, they require excessive memory usage to store the optimization states for training. To alleviate this, the low-rank adaptation (LoRA) is proposed to reduce the optimization states by training fewer parameters. However, LoRA restricts overall weight update matrices to be low-rank, limiting the model performance. In this work, we investigate the dynamics of LoRA and identify that it can be approximated by a random projection. Based on this observation, we propose Flora, which is able to achieve high-rank updates by resampling the projection matrices while enjoying the sublinear space complexity of optimization states. We conduct experiments across different tasks and model architectures to verify the effectiveness of our approach.

翻译：尽管大型神经网络在完成不同任务时展现出卓越能力，但其训练过程中存储优化状态需消耗大量内存。为缓解这一问题，低秩适配（LoRA）通过训练更少参数来减少优化状态。然而，LoRA将整体权重更新矩阵限制为低秩，从而限制了模型性能。本研究通过探究LoRA的动态特性，发现其可近似为随机投影。基于这一发现，我们提出Flora方法，该方法通过重采样投影矩阵实现高秩更新，同时保持优化状态的亚线性空间复杂度。我们在不同任务和模型架构上开展实验，验证了本方法的有效性。