Low-rank adaptations (LoRA) are often employed to fine-tune large language models (LLMs) for new tasks. This paper investigates LoRA composability for cross-task generalization and introduces LoraHub, a simple framework devised for the purposive assembly of LoRA modules trained on diverse given tasks, with the objective of achieving adaptable performance on unseen tasks. With just a few examples from a new task, LoraHub can fluidly combine multiple LoRA modules, eliminating the need for human expertise and assumptions. Notably, the composition requires neither additional model parameters nor gradients. Empirical results on the Big-Bench Hard benchmark suggest that LoraHub, while not surpassing the performance of in-context learning, offers a notable performance-efficiency trade-off in few-shot scenarios by employing a significantly reduced number of tokens per example during inference. Notably, LoraHub establishes a better upper bound compared to in-context learning when paired with different demonstration examples, demonstrating its potential for future development. Our vision is to establish a platform for LoRA modules, empowering users to share their trained LoRA modules. This collaborative approach facilitates the seamless application of LoRA modules to novel tasks, contributing to an adaptive ecosystem. Our code is available at https://github.com/sail-sg/lorahub, and all the pre-trained LoRA modules are released at https://huggingface.co/lorahub.

翻译：低秩自适应（LoRA）通常用于对新任务进行大语言模型（LLM）的微调。本文研究了LoRA在跨任务泛化中的可组合性，并介绍了LoraHub——一个为有目的地组合在不同给定任务上训练得到的LoRA模块而设计的简单框架，旨在实现对未见任务的适应性性能。仅需来自新任务的少量示例，LoraHub即可灵活组合多个LoRA模块，无需依赖人类专业知识或预设假设。值得注意的是，该组合过程既不引入额外的模型参数，也不涉及梯度计算。在Big-Bench Hard基准测试上的实证结果表明，尽管LoraHub的性能未超越上下文学习，但其在少样本场景中通过每个示例在推理时使用显著更少的标记数，提供了优异的性能-效率权衡。特别地，当与不同的演示示例结合时，LoraHub相比上下文学习建立了更优的性能上限，展现了其未来发展的潜力。我们的愿景是建立一个LoRA模块共享平台，使用户能够分享其训练好的LoRA模块。这种协作方式有助于将LoRA模块无缝应用于新任务，从而构建一个自适应的生态系统。我们的代码发布于https://github.com/sail-sg/lorahub，所有预训练的LoRA模块发布于https://huggingface.co/lorahub。