Knowledge editing aims to inject knowledge updates into language models to keep them correct and up-to-date. However, its current evaluation strategies are notably impractical: they solely update with well-curated structured facts (triplets with subjects, relations, and objects), whereas real-world knowledge updates commonly emerge in unstructured texts like news articles. In this paper, we propose a new benchmark, Unstructured Knowledge Editing (UKE). It evaluates editing performance directly using unstructured texts as knowledge updates, termed unstructured facts. Hence UKE avoids the laborious construction of structured facts and enables efficient and responsive knowledge editing, becoming a more practical benchmark. We conduct extensive experiments on newly built datasets and demonstrate that UKE poses a significant challenge to state-of-the-art knowledge editing methods, resulting in their critical performance declines. We further show that this challenge persists even if we extract triplets as structured facts. Our analysis discloses key insights to motivate future research in UKE for more practical knowledge editing.

翻译：知识编辑旨在向语言模型中注入知识更新，以保持其正确性和时效性。然而，当前的评估策略存在明显的实用性不足：它们仅使用精心整理的结构化事实（包含主体、关系和客体的三元组）进行更新，而现实世界中的知识更新通常以非结构化文本（如新闻文章）的形式出现。在本文中，我们提出一个新的基准——非结构化知识编辑（UKE）。该基准直接使用非结构化文本作为知识更新（称为非结构化事实）来评估编辑性能，从而避免了结构化事实的繁琐构建，实现了高效且响应迅速的知识编辑，成为一个更实用的基准。我们在新构建的数据集上进行了大量实验，结果表明UKE对当前最先进的知识编辑方法构成了重大挑战，导致其性能显著下降。我们进一步证明，即使将非结构化事实提取为三元组形式的结构化事实，这一挑战依然存在。我们的分析揭示了关键见解，以激励未来在UKE方向的研究，从而推动更实用的知识编辑。