Model editing aims to correct outdated or erroneous knowledge in large models without costly retraining. Recent research discovered that the mid-layer representation of the subject's final token in a prompt has a strong influence on factual predictions, and developed Large Language Model (LLM) editing techniques based on this observation. However, for Vision-LLMs (VLLMs), how visual representations impact the predictions from a decoder-only language model remains largely unexplored. To the best of our knowledge, model editing for VLLMs has not been extensively studied in the literature. In this work, we employ the contribution allocation and noise perturbation methods to measure the contributions of visual representations for token predictions. Our attribution analysis shows that visual representations in mid-to-later layers that are highly relevant to the prompt contribute significantly to predictions. Based on these insights, we propose VisEdit, a novel model editor for VLLMs that effectively corrects knowledge by editing intermediate visual representations in regions important to the edit prompt. We evaluated VisEdit using multiple VLLM backbones and public VLLM editing benchmark datasets. The results show the superiority of VisEdit over the strong baselines adapted from existing state-of-the-art editors for LLMs.

翻译：模型编辑旨在无需昂贵重新训练的情况下修正大型模型中的过时或错误知识。近期研究发现，提示中主体最终标记的中间层表示对事实预测具有显著影响，并基于此观察开发了大型语言模型（LLM）编辑技术。然而，对于视觉语言模型（VLLMs），视觉表示如何影响仅解码器语言模型的预测在很大程度上仍未得到探索。据我们所知，针对VLLMs的模型编辑在现有文献中尚未得到广泛研究。本工作中，我们采用贡献分配与噪声扰动方法来度量视觉表示对标记预测的贡献度。归因分析表明，与提示高度相关的中后层视觉表示对预测具有重要贡献。基于这些发现，我们提出了VisEdit——一种针对VLLMs的新型模型编辑器，通过编辑对编辑提示重要区域的中间视觉表示来有效校正知识。我们使用多种VLLM骨干网络和公开的VLLM编辑基准数据集对VisEdit进行了评估。结果表明，相较于从现有最先进LLM编辑器改编的强基线方法，VisEdit展现出显著优势。