The widespread availability of fine-tuned LoRA modules for open pre-trained models has led to an interest in methods that can adaptively merge LoRAs to improve performance. These methods typically include some way of selecting LoRAs from a pool and tune merging coefficients based on a task-specific dataset. While adaptive merging methods have demonstrated improvements in some settings, no past work has attempted to recycle LoRAs found "in the wild" on model repositories like the Hugging Face Hub. To address this gap, we consider recycling from a pool of nearly 1,000 user-contributed LoRAs trained from the Llama 3.1 8B-Instruct language model. Our empirical study includes a range of adaptive and non-adaptive merging methods in addition to a new method designed via a wide search over the methodological design space. We demonstrate that adaptive merging methods can improve performance over the base model but provide limited benefit over training a new LoRA on the same data used to set merging coefficients. We additionally find not only that the specific choice of LoRAs to merge has little importance, but that using LoRAs with randomly initialized parameter values yields similar performance. This raises the possibility that adaptive merging from recycled LoRAs primarily works via some kind of regularization effect, rather than by enabling positive cross-task transfer. To better understand why past work has proven successful, we confirm that positive transfer is indeed possible when there are highly relevant LoRAs in the pool. We release the model checkpoints and code online.

翻译：针对开放预训练模型的微调LoRA模块的广泛可用性，引发了人们对能够自适应合并LoRA以提升性能方法的兴趣。这些方法通常包括从池中选择LoRA的方式，并基于特定任务数据集调整合并系数。尽管自适应合并方法在某些场景下已展现出性能提升，但以往工作尚未尝试回收在模型仓库（如Hugging Face Hub）中"野生"发现的LoRA。为填补这一空白，我们考虑从近1,000个基于Llama 3.1 8B-Instruct语言模型训练的用户贡献LoRA池中进行回收。我们的实证研究涵盖了多种自适应与非自适应合并方法，还包括一种通过对方法设计空间进行广泛搜索而设计的新方法。我们证明，自适应合并方法可以提升基础模型的性能，但与使用相同数据训练新LoRA相比，其优势有限。此外，我们发现不仅待合并LoRA的具体选择影响甚微，而且使用随机初始化参数值的LoRA也能产生相似性能。这提出了一个可能性：从回收LoRA进行自适应合并主要通过某种正则化效应发挥作用，而非通过实现正向跨任务迁移。为更好理解以往工作成功的原因，我们证实当池中存在高度相关的LoRA时，正向迁移确实可能发生。我们已在线发布模型检查点与代码。