Low-rank adaptations (LoRA) are often employed to fine-tune large language models (LLMs) for new tasks. This paper investigates LoRA composability for cross-task generalization and introduces LoraHub, a strategic framework devised for the purposive assembly of LoRA modules trained on diverse given tasks, with the objective of achieving adaptable performance on unseen tasks. With just a few examples from a novel task, LoraHub enables the fluid combination of multiple LoRA modules, eradicating the need for human expertise. Notably, the composition requires neither additional model parameters nor gradients. Our empirical results, derived from the Big-Bench Hard (BBH) benchmark, suggest that LoraHub can effectively mimic the performance of in-context learning in few-shot scenarios, excluding the necessity of in-context examples alongside each inference input. A significant contribution of our research is the fostering of a community for LoRA, where users can share their trained LoRA modules, thereby facilitating their application to new tasks. We anticipate this resource will widen access to and spur advancements in general intelligence as well as LLMs in production. Code will be available at https://github.com/sail-sg/lorahub.

翻译：低秩适配（LoRA）常被用于微调大型语言模型以完成新任务。本文研究LoRA在跨任务泛化中的可组合性，并提出LoraHub——一个为有目的地组装基于不同任务训练的LoRA模块而设计的策略框架，旨在实现对未见任务的适应性性能。仅需少量来自新任务的示例，LoraHub即可实现多个LoRA模块的灵活组合，无需人工干预。值得注意的是，该组合既不需要额外模型参数，也不依赖梯度计算。我们在Big-Bench Hard（BBH）基准上的实证结果表明，LoraHub能够在少样本场景下有效模拟上下文学习的性能，且无需在每次推理输入中附带上下文示例。本研究的重要贡献在于构建了一个LoRA社区生态，用户可共享其训练的LoRA模块，从而促进其在新任务中的应用。我们预期该资源将拓宽通用智能及生产级大型语言模型的获取渠道并推动其发展。代码将发布于https://github.com/sail-sg/lorahub。