Model merging has emerged as a crucial technique in Deep Learning, enabling the integration of multiple models into a unified system while preserving performance and scalability. In this respect, the compositional properties of low-rank adaptation techniques (e.g., LoRA) have proven beneficial, as simple averaging LoRA modules yields a single model that mostly integrates the capabilities of all individual modules. Building on LoRA, we take a step further by imposing that the merged model matches the responses of all learned modules. Solving this objective in closed form yields an indeterminate system with A and B as unknown variables, indicating the existence of infinitely many closed-form solutions. To address this challenge, we introduce LoRM, an alternating optimization strategy that trains one LoRA matrix at a time. This allows solving for each unknown variable individually, thus finding a unique solution. We apply our proposed methodology to Federated Class-Incremental Learning (FCIL), ensuring alignment of model responses both between clients and across tasks. Our method demonstrates state-of-the-art performance across a range of FCIL scenarios.

翻译：模型合并已成为深度学习中的一项关键技术，能够将多个模型集成到统一系统中，同时保持性能和可扩展性。在这方面，低秩自适应技术（如LoRA）的组合特性已被证明具有优势，因为简单平均LoRA模块可产生一个基本整合所有独立模块能力的单一模型。基于LoRA，我们进一步要求合并模型匹配所有已学习模块的响应。以闭式求解该目标会产生一个以A和B为未知变量的不定系统，表明存在无限多个闭式解。为应对这一挑战，我们提出LoRM——一种交替优化策略，每次仅训练一个LoRA矩阵。这使得可以单独求解每个未知变量，从而找到唯一解。我们将所提方法应用于联邦类增量学习（FCIL），确保模型响应在客户端间和跨任务间均保持一致。我们的方法在一系列FCIL场景中展现了最先进的性能。