Low-Rank Adaptation (LoRA) has achieved remarkable training results by freezing the original weights and training only low-rank matrices, establishing itself as the predominant fine-tuning method for LLMs. In pursuit of performance closer to full-parameter training, a series of LoRA variants have emerged, such as LoRA+, PISSA, Olora, and LoRA-GA. However, these improvements complicate the initial setup of model training and increase initialization time. More importantly, they overlook the internal interactions of the original weight information. To address these issues, we introduce a novel theory, ``Weight Guide'' aimed at continuously guiding trainable matrices through the original weights during training to enhance the utilization of weight information. Based on this theory, we designed a new PEFT technique called Bone (\textbf{B}l\textbf{o}ck Affi\textbf{ne}), which not only enhances the utilization of original weight information but also emphasizes the internal connections between weights, leading to faster convergence and better data fitting. Experimental comparisons across two different LLM architectures (LLaMA2, RWKV6) and various parameter scales demonstrate that the Bone structure can achieve rapid convergence and superior data fitting without the need for complex initialization. For example, when fine-tuning LLaMA2-7B on the MetaMathQA dataset and validating on GSM8k and math benchmarks, Bone achieved fine-tuning scores of 49.36 and 8.8, respectively, outperforming PISSA by 5.84\% and 1.96\%.

翻译：低秩适应（LoRA）通过冻结原始权重并仅训练低秩矩阵，取得了显著的训练效果，已成为大语言模型（LLM）的主流微调方法。为追求更接近全参数训练的性能，一系列LoRA变体相继涌现，如LoRA+、PISSA、Olora和LoRA-GA。然而，这些改进使模型训练的初始设置复杂化并增加了初始化时间。更重要的是，它们忽视了原始权重信息的内部交互作用。为解决这些问题，我们提出了一种新颖的理论——“权重引导”，旨在训练过程中通过原始权重持续引导可训练矩阵，以提升权重信息的利用率。基于此理论，我们设计了一种名为Bone（\textbf{B}l\textbf{o}ck Affi\textbf{ne}）的新型参数高效微调（PEFT）技术，它不仅增强了对原始权重信息的利用，还强调了权重间的内部关联，从而实现了更快的收敛速度和更好的数据拟合能力。在两种不同LLM架构（LLaMA2、RWKV6）及多种参数规模上的实验对比表明，Bone结构无需复杂初始化即可实现快速收敛和优异的数据拟合效果。例如，在MetaMathQA数据集上微调LLaMA2-7B模型，并在GSM8k和数学基准测试上进行验证时，Bone分别取得了49.36和8.8的微调分数，较PISSA提升了5.84%和1.96%。