Large language models (LLMs) have played a pivotal role in building communicative AI to imitate human behaviors but face the challenge of efficient customization. To tackle this challenge, recent studies have delved into the realm of model editing, which manipulates specific memories of language models and changes the related language generation. However, the robustness of model editing remains an open question. This work seeks to understand the strengths and limitations of editing methods, thus facilitating robust, realistic applications of communicative AI. Concretely, we conduct extensive analysis to address the three key research questions. Q1: Can edited LLMs behave consistently resembling communicative AI in realistic situations? Q2: To what extent does the rephrasing of prompts lead LLMs to deviate from the edited knowledge memory? Q3: Which knowledge features are correlated with the performance and robustness of editing? Our experimental results uncover a substantial disparity between existing editing methods and the practical application of LLMs. On rephrased prompts that are complex and flexible but common in realistic applications, the performance of editing experiences a significant decline. Further analysis shows that more popular knowledge is memorized better, easier to recall, and more challenging to edit effectively.

翻译：大型语言模型（LLMs）在构建模仿人类行为的交际型人工智能中发挥了关键作用，但面临着高效定制的挑战。为应对这一挑战，近期研究深入探索了模型编辑领域，通过操控语言模型的特定记忆来改变相关语言生成。然而，模型编辑的稳健性仍是一个未解之谜。本研究旨在理解编辑方法的优势与局限性，从而促进交际型人工智能的稳健、现实应用。具体而言，我们进行了广泛分析以解决三个关键研究问题。问题1：经过编辑的大型语言模型能否在现实情境中持续保持类似交际型人工智能的行为一致性？问题2：提示文本的改写会在多大程度上导致大型语言模型偏离编辑后的知识记忆？问题3：哪些知识特征与编辑的性能和稳健性相关？我们的实验结果表明，现有编辑方法与大型语言模型的实际应用之间存在显著差距。面对现实应用中常见但复杂灵活的改写提示，编辑的性能显著下降。进一步分析显示，更流行的知识记忆更牢固、更容易被回忆起，也更难被有效编辑。