The rapid growth of large models has raised concerns about their environmental impact and equity in accessibility due to significant computational costs. Low-Rank Adapters (LoRA) offer a lightweight solution for finetuning large models, resulting in an abundance of publicly available adapters tailored to diverse domains. We ask: Can these pretrained adapters be leveraged to further streamline adaptation to new tasks while addressing these challenges? We introduce EigenLoRAx, a parameter-efficient finetuning method that recycles existing adapters to create a principal subspace aligned with their shared domain knowledge which can be further augmented with orthogonal basis vectors in low-resource scenarios. This enables rapid adaptation to new tasks by learning only lightweight coefficients on the principal components of the subspace-eliminating the need to finetune entire adapters. EigenLoRAx requires significantly fewer parameters and memory, improving efficiency for both training and inference. Our method demonstrates strong performance across diverse domains and tasks, offering a scalable for edge-based applications, personalization, and equitable deployment of large models in resource-constrained environments.

翻译：大型模型的快速增长因其高昂的计算成本引发了对其环境影响及可访问性公平性的担忧。低秩适配器（LoRA）为微调大型模型提供了一种轻量级解决方案，从而产生了大量针对不同领域的公开适配器。我们提出疑问：能否利用这些预训练适配器来进一步简化新任务的适应过程，同时应对这些挑战？本文提出EigenLoRAx，这是一种参数高效的微调方法，通过回收现有适配器构建与其共享领域知识对齐的主成分子空间，并可在低资源场景下通过正交基向量进行扩展。该方法通过仅学习子空间主成分上的轻量级系数，即可快速适应新任务，无需对整个适配器进行微调。EigenLoRAx显著减少了参数数量和内存占用，提升了训练与推理效率。我们的方法在多个领域和任务中均表现出优异性能，为基于边缘的应用、个性化部署以及在资源受限环境中公平部署大型模型提供了可扩展的解决方案。