Low-rank adaptations (LoRA) are often employed to fine-tune large language models (LLMs) for new tasks. This paper investigates LoRA composability for cross-task generalization and introduces LoraHub, a simple framework devised for the purposive assembly of LoRA modules trained on diverse given tasks, with the objective of achieving adaptable performance on unseen tasks. With just a few examples from a new task, LoraHub can fluidly combine multiple LoRA modules, eliminating the need for human expertise and assumptions. Notably, the composition requires neither additional model parameters nor gradients. Empirical results on the Big-Bench Hard benchmark suggest that LoraHub, while not surpassing the performance of in-context learning, offers a notable performance-efficiency trade-off in few-shot scenarios by employing a significantly reduced number of tokens per example during inference. Notably, LoraHub establishes a better upper bound compared to in-context learning when paired with different demonstration examples, demonstrating its potential for future development. Our vision is to establish a platform for LoRA modules, empowering users to share their trained LoRA modules. This collaborative approach facilitates the seamless application of LoRA modules to novel tasks, contributing to an adaptive ecosystem. Our code is available at https://github.com/sail-sg/lorahub, and all the pre-trained LoRA modules are released at https://huggingface.co/lorahub.

翻译：低秩适配（LoRA）常被用于微调大型语言模型（LLMs）以适应新任务。本文探究了LoRA在跨任务泛化中的可组合性，并提出了LoraHub——一个简单框架，旨在有目的地组合在多种给定任务上训练的LoRA模块，以实现对未见任务的适应性性能。仅需来自新任务的少量样本，LoraHub即可流畅地组合多个LoRA模块，无需人工专业知识或假设。值得注意的是，该组合过程既不需要额外模型参数，也不依赖梯度计算。在Big-Bench Hard基准上的实验结果表明，尽管LoraHub未能超越上下文学习的性能，但在小样本场景中通过显著减少每个示例的推理令牌数，实现了性能与效率的显著平衡。尤为关键的是，当搭配不同示范样本时，LoraHub相比上下文学习建立了更优的性能上限，展现了其未来发展的潜力。我们的愿景是构建一个LoRA模块平台，使用户能够共享其训练好的LoRA模块。这种协作方式将促进LoRA模块无缝应用于新任务，助力形成自适应生态系统。相关代码已开源至https://github.com/sail-sg/lorahub，所有预训练LoRA模块可在https://huggingface.co/lorahub获取。