Low-Rank Adaptation (LoRA) has emerged as a popular technique for fine-tuning large language models (LLMs) to various domains due to its modular design and widespread availability on platforms like Huggingface. This modularity has sparked interest in combining multiple LoRAs to enhance LLM capabilities. However, existing methods for LoRA composition primarily focus on task-specific adaptations that require additional training, and current model merging techniques often fail to fully leverage LoRA's modular nature, leading to parameter interference and performance degradation. In this paper, we investigate the feasibility of disassembling and reassembling multiple LoRAs at a finer granularity, analogous to assembling LEGO blocks. We introduce the concept of Minimal Semantic Units (MSUs), where the parameters corresponding to each rank in LoRA function as independent units. These MSUs demonstrate permutation invariance and concatenation-summation equivalence properties, enabling flexible combinations to create new LoRAs. Building on these insights, we propose the LoRA-LEGO framework. This framework conducts rank-wise parameter clustering by grouping MSUs from different LoRAs into $k$ clusters. The centroid of each cluster serves as a representative MSU, enabling the assembly of a merged LoRA with an adjusted rank of $k$. Additionally, we apply a dual reweighting strategy to optimize the scale of the merged LoRA. Experiments across various benchmarks demonstrate that our method outperforms existing approaches in LoRA merging.

翻译：低秩适配（LoRA）因其模块化设计以及在Huggingface等平台上的广泛可用性，已成为将大语言模型（LLM）微调至不同领域的主流技术。这种模块化特性激发了组合多个LoRA以增强LLM能力的兴趣。然而，现有的LoRA组合方法主要集中于需要额外训练的任务特定适配，且当前的模型融合技术往往未能充分利用LoRA的模块化本质，导致参数干扰和性能下降。本文探讨了以更细粒度拆解与重组多个LoRA的可行性，其过程类似于拼搭乐高积木。我们引入了最小语义单元（MSU）的概念，其中LoRA中每个秩对应的参数作为独立单元运作。这些MSU展现出排列不变性与拼接-求和等价性，能够灵活组合以构建新的LoRA。基于这些发现，我们提出了LoRA-LEGO框架。该框架通过对来自不同LoRA的MSU进行分组，将其聚类为$k$个簇，实现秩间参数聚类。每个簇的质心作为代表性MSU，从而能够组装出一个调整后秩为$k$的融合LoRA。此外，我们采用双重重加权策略以优化融合LoRA的尺度。在多个基准测试上的实验表明，我们的方法在LoRA融合任务上优于现有技术。