Model editing aims to data-efficiently correct predictive errors of large pre-trained models while ensuring generalization to neighboring failures and locality to minimize unintended effects on unrelated examples. While significant progress has been made in editing Transformer-based large language models, effective strategies for editing vision Transformers (ViTs) in computer vision remain largely untapped. In this paper, we take initial steps towards correcting predictive errors of ViTs, particularly those arising from subpopulation shifts. Taking a locate-then-edit approach, we first address the where-to-edit challenge by meta-learning a hypernetwork on CutMix-augmented data generated for editing reliability. This trained hypernetwork produces generalizable binary masks that identify a sparse subset of structured model parameters, responsive to real-world failure samples. Afterward, we solve the how-to-edit problem by simply fine-tuning the identified parameters using a variant of gradient descent to achieve successful edits. To validate our method, we construct an editing benchmark that introduces subpopulation shifts towards natural underrepresented images and AI-generated images, thereby revealing the limitations of pre-trained ViTs for object recognition. Our approach not only achieves superior performance on the proposed benchmark but also allows for adjustable trade-offs between generalization and locality. Our code is available at https://github.com/hustyyq/Where-to-Edit.

翻译：模型编辑旨在数据高效地修正大型预训练模型的预测错误，同时确保对邻近故障的泛化能力与局部性，以最小化对无关样本的意外影响。尽管在编辑基于Transformer的大型语言模型方面已取得显著进展，但在计算机视觉领域编辑视觉Transformer（ViT）的有效策略仍基本未被探索。本文中，我们迈出了修正ViT预测错误的第一步，特别是那些由子群体偏移引起的错误。采用“定位-编辑”方法，我们首先通过元学习一个超网络来解决“何处编辑”的挑战，该超网络在专为编辑可靠性生成的CutMix增强数据上进行训练。训练后的超网络可生成可泛化的二值掩码，以识别对现实世界故障样本响应的结构化模型参数的稀疏子集。随后，我们通过简单使用梯度下降变体微调已识别的参数来解决“如何编辑”问题，从而实现成功编辑。为验证我们的方法，我们构建了一个编辑基准，该基准引入了面向自然欠表征图像和AI生成图像的子群体偏移，从而揭示了预训练ViT在物体识别方面的局限性。我们的方法不仅在所提出的基准上实现了优越性能，还允许在泛化性与局部性之间进行可调节的权衡。我们的代码可在https://github.com/hustyyq/Where-to-Edit获取。