Low-Rank Adaptation (LoRA) is a widely used Parameter-Efficient Fine-Tuning (PEFT) method that updates an initial weight matrix $W_0$ with a delta matrix $\Delta W$ consisted by two low-rank matrices $A$ and $B$. A previous study suggested that there is correlation between $W_0$ and $\Delta W$. In this study, we aim to delve deeper into relationships between $W_0$ and low-rank matrices $A$ and $B$ to further comprehend the behavior of LoRA. In particular, we analyze a conversion matrix that transform $W_0$ into low-rank matrices, which encapsulates information about the relationships. Our analysis reveals that the conversion matrices are similar across each layer. Inspired by these findings, we hypothesize that a single linear layer, which takes each layer's $W_0$ as input, can yield task-adapted low-rank matrices. To confirm this hypothesis, we devise a method named Conditionally Parameterized LoRA (CondLoRA) that updates initial weight matrices with low-rank matrices derived from a single linear layer. Our empirical results show that CondLoRA maintains a performance on par with LoRA, despite the fact that the trainable parameters of CondLoRA are fewer than those of LoRA. Therefore, we conclude that "a single linear layer yields task-adapted low-rank matrices."

翻译：低秩适配（LoRA）是一种广泛使用的参数高效微调（PEFT）方法，通过由两个低秩矩阵 $A$ 和 $B$ 构成的增量矩阵 $\Delta W$ 来更新初始权重矩阵 $W_0$。此前研究表明 $W_0$ 与 $\Delta W$ 之间存在相关性。本研究旨在深入探究 $W_0$ 与低秩矩阵 $A$、$B$ 之间的关系，以进一步理解LoRA的行为特性。具体而言，我们分析了将 $W_0$ 转换为低秩矩阵的转换矩阵，该矩阵蕴含了关于上述关系的信息。分析表明，各层转换矩阵具有相似性。基于这些发现，我们提出假设：以每层 $W_0$ 作为输入的单一线性层即可生成任务自适应低秩矩阵。为验证该假设，我们设计了条件参数化LoRA（CondLoRA）方法，通过单一线性层推导出的低秩矩阵来更新初始权重矩阵。实验结果表明，尽管CondLoRA的可训练参数少于LoRA，但其性能与LoRA相当。因此，我们得出结论："一条线性层即可产生任务自适应低秩矩阵"。