Parameter-Efficient Fine-Tuning (PEFT) methods enable efficient adaptation of Large Language Models (LLMs) to various downstream applications. However, the effectiveness of the PEFT diminishes notably when downstream tasks require accurate learning of factual knowledge. In this paper, we adopt a semantic perspective to investigate this phenomenon, uncovering the reasons behind PEFT's limitations in knowledge learning task. Our findings reveal that: (1) PEFT presents a notable risk of pushing the model away from the intended knowledge target; (2) multiple knowledge interfere with each other, and such interference suppresses the learning and expression of knowledge features. Based on these insights, we introduce a data filtering strategy to exclude data that is detrimental to knowledge learning and a re-weighted learning strategy to make the model attentive to semantic distance during knowledge learning. Experimental results demonstrate the effectiveness of the proposed method on open-source large language model, further validate the semantic challenge in PEFT, thus paving the way for future research.

翻译：参数高效微调（PEFT）方法能够使大型语言模型（LLM）高效适应各种下游应用。然而，当下游任务需要准确学习事实性知识时，PEFT的有效性会显著下降。本文从语义视角出发探究这一现象，揭示了PEFT在知识学习任务中存在局限性的原因。我们的研究发现：（1）PEFT存在使模型偏离预期知识目标的显著风险；（2）多项知识会相互干扰，这种干扰抑制了知识特征的学习与表达。基于这些发现，我们提出了一种数据过滤策略以排除不利于知识学习的数据，以及一种重加权学习策略使模型在知识学习过程中关注语义距离。实验结果表明，所提方法在开源大型语言模型上具有有效性，进一步验证了PEFT面临的语义挑战，从而为未来研究铺平了道路。