We present Generalized LoRA (GLoRA), an advanced approach for universal parameter-efficient fine-tuning tasks. Enhancing Low-Rank Adaptation (LoRA), GLoRA employs a generalized prompt module to optimize pre-trained model weights and adjust intermediate activations, providing more flexibility and capability across diverse tasks and datasets. Moreover, GLoRA facilitates efficient parameter adaptation by employing a scalable, modular, layer-wise structure search that learns individual adapter of each layer. Originating from a unified mathematical formulation, GLoRA exhibits strong transfer learning, few-shot learning and domain generalization abilities, as it adapts to new tasks through not only weights but also additional dimensions like activations. Comprehensive experiments demonstrate that GLoRA outperforms all previous methods in natural, specialized, and structured vision benchmarks, achieving superior accuracy with fewer parameters and computations. The proposed method on LLaMA-1 and LLaMA-2 also show considerable enhancements compared to the original LoRA in the language domain. Furthermore, our structural re-parameterization design ensures that GLoRA incurs no extra inference cost, rendering it a practical solution for resource-limited applications. Code and models are available at: https://github.com/Arnav0400/ViT-Slim/tree/master/GLoRA.

翻译：我们提出了广义LoRA（GLoRA），一种面向通用参数高效微调任务的先进方法。通过增强低秩适应（LoRA），GLoRA采用通用提示模块来优化预训练模型权重并调整中间激活值，在多样化任务与数据集上展现出更强的灵活性与能力。此外，GLoRA通过可扩展、模块化、逐层解耦的结构搜索机制，为每层学习独立的适配器，从而实现高效的参数适配。基于统一的数学公式，GLoRA不仅通过权重调整适应新任务，还通过激活值等额外维度进行适应，展现出强大的迁移学习、少样本学习及领域泛化能力。全面实验表明，GLoRA在自然、专业及结构化视觉基准任务上均优于以往所有方法，以更少的参数和计算量实现了更高的准确率。在LLaMA-1和LLaMA-2上的实验结果显示，该方法在语言领域相比原始LoRA也有显著提升。此外，我们采用的结构重参数化设计确保GLoRA在推理阶段无额外计算开销，使其成为资源受限场景下的实用解决方案。代码与模型已开源至：https://github.com/Arnav0400/ViT-Slim/tree/master/GLoRA。