Low-Rank Adaptation (LoRA) has emerged as a parameter-efficient approach for fine-tuning large language models. However, conventional LoRA adapters are typically trained for a single task, limiting their applicability in real-world settings where inputs may span diverse and unpredictable domains. At inference time, existing approaches combine multiple LoRAs for improving performance on diverse tasks, while usually requiring labeled data or additional task-specific training, which is expensive at scale. In this work, we introduce LoRA on the Go (LoGo), a training-free framework that dynamically selects and merges adapters at the instance level without any additional requirements. LoGo leverages signals extracted from a single forward pass through LoRA adapters, to identify the most relevant adapters and determine their contributions on-the-fly. Across 5 NLP benchmarks, 27 datasets, and 3 model families, LoGo outperforms training-based baselines on some tasks upto a margin of 3.6% while remaining competitive on other tasks and maintaining inference throughput, highlighting its effectiveness and practicality.

翻译：低秩适配（LoRA）已成为微调大语言模型的一种参数高效方法。然而，传统的LoRA适配器通常针对单一任务进行训练，这限制了其在现实场景中的应用，因为输入可能涉及多样且不可预测的领域。在推理阶段，现有方法通过组合多个LoRA适配器来提升多样化任务的性能，但通常需要标注数据或额外的任务特定训练，这在规模化应用中成本高昂。本文提出LoRA动态适配（LoGo），一种无需训练的框架，能够在实例级别动态选择和融合适配器，无需任何额外条件。LoGo利用从LoRA适配器单次前向传播中提取的信号，实时识别最相关的适配器并确定其贡献。在5个自然语言处理基准测试、27个数据集和3个模型系列上的实验表明，LoGo在某些任务上优于基于训练的基线方法，最高提升3.6%，同时在其他任务上保持竞争力，并维持推理吞吐量，突显了其有效性和实用性。