Model editing techniques modify a minor proportion of knowledge in Large Language Models (LLMs) at a relatively low cost, which have demonstrated notable success. Existing methods assume Transformer Layer (TL) hidden states are values of key-value memories of the Feed-Forward Network (FFN). They usually optimize the TL hidden states to memorize target knowledge and use it to update the weights of the FFN in LLMs. However, the information flow of TL hidden states comes from three parts: Multi-Head Self-Attention (MHSA), FFN, and residual connections. Existing methods neglect the fact that the TL hidden states contains information not specifically required for FFN. Consequently, the performance of model editing decreases. To achieve more precise model editing, we analyze hidden states of MHSA and FFN, finding that MHSA encodes certain general knowledge extraction patterns. This implies that MHSA weights do not require updating when new knowledge is introduced. Based on above findings, we introduce PMET, which simultaneously optimizes Transformer Component (TC, namely MHSA and FFN) hidden states, while only using the optimized TC hidden states of FFN to precisely update FFN weights. Our experiments demonstrate that PMET exhibits state-of-the-art performance on both the \textsc{counterfact} and zsRE datasets. Our ablation experiments substantiate the effectiveness of our enhancements, further reinforcing the finding that the MHSA encodes certain general knowledge extraction patterns and indicating its storage of a small amount of factual knowledge. Our code is available at \url{https://github.com/xpq-tech/PMET.git}.

翻译：[translated abstract in Chinese] 模型编辑技术以较低成本修改大型语言模型（LLMs）中的少量知识，已展现出显著成效。现有方法假设Transformer层（TL）的隐藏状态是前馈网络（FFN）键值记忆的值，通常通过优化TL隐藏状态以记忆目标知识，并据此更新LLMs中FFN的权重。然而，TL隐藏状态的信息流来源于三部分：多头自注意力机制（MHSA）、FFN及残差连接。现有方法忽视了TL隐藏状态包含FFN非必要信息的特性，导致模型编辑性能下降。为实现更精准的模型编辑，本文分析了MHSA与FFN的隐藏状态，发现MHSA编码了某些通用知识提取模式——这意味着引入新知识时无需更新MHSA权重。基于此发现，我们提出PMET方法，同步优化Transformer组件（TC，即MHSA与FFN）的隐藏状态，并仅利用优化后的FFN的TC隐藏状态精确更新FFN权重。实验表明，PMET在\textsc{counterfact}与zsRE数据集上均达到最优性能。消融实验验证了所提改进的有效性，进一步证实MHSA编码通用知识提取模式并存储少量事实知识的发现。代码开源地址：\url{https://github.com/xpq-tech/PMET.git}