We present Generalized LoRA (GLoRA), an advanced approach for universal parameter-efficient fine-tuning tasks. Enhancing Low-Rank Adaptation (LoRA), GLoRA employs a generalized prompt module to optimize pre-trained model weights and adjust intermediate activations, providing more flexibility and capability across diverse tasks and datasets. Moreover, GLoRA facilitates efficient parameter adaptation by employing a scalable, modular, layer-wise structure search that learns individual adapter of each layer. Originating from a unified mathematical formulation, GLoRA exhibits strong transfer learning, few-shot learning and domain generalization abilities, as it adjusts to new tasks through additional dimensions on weights and activations. Comprehensive experiments demonstrate that GLoRA outperforms all previous methods in natural, specialized, and structured benchmarks, achieving superior accuracy with fewer parameters and computations on various datasets. Furthermore, our structural re-parameterization design ensures that GLoRA incurs no extra inference cost, rendering it a practical solution for resource-limited applications. Code is available at: https://github.com/Arnav0400/ViT-Slim/tree/master/GLoRA.

翻译：我们提出广义LoRA（GLoRA），这是一种针对通用参数高效微调任务的高级方法。通过增强低秩适应（LoRA），GLoRA采用广义提示模块来优化预训练模型权重并调整中间激活，从而在多样化的任务和数据集中提供更高的灵活性和能力。此外，GLoRA通过采用可扩展、模块化、逐层结构搜索来学习每层的独立适配器，从而促进高效的参数适应。源于统一的数学公式，GLoRA展现出强大的迁移学习、少样本学习和领域泛化能力，因为它通过权重和激活的附加维度来适应新任务。全面的实验表明，在自然、专业和结构化基准测试中，GLoRA在所有先前方法中表现优异，在各种数据集上以更少的参数和计算量实现了更高的准确率。此外，我们的结构重参数化设计确保了GLoRA在推理过程中不引入额外成本，使其成为资源受限应用中的实用解决方案。代码可从以下地址获取：https://github.com/Arnav0400/ViT-Slim/tree/master/GLoRA。