We revisit and advance visual prompting (VP), an input prompting technique for vision tasks. VP can reprogram a fixed, pre-trained source model to accomplish downstream tasks in the target domain by simply incorporating universal prompts (in terms of input perturbation patterns) into downstream data points. Yet, it remains elusive why VP stays effective even given a ruleless label mapping (LM) between the source classes and the target classes. Inspired by the above, we ask: How is LM interrelated with VP? And how to exploit such a relationship to improve its accuracy on target tasks? We peer into the influence of LM on VP and provide an affirmative answer that a better 'quality' of LM (assessed by mapping precision and explanation) can consistently improve the effectiveness of VP. This is in contrast to the prior art where the factor of LM was missing. To optimize LM, we propose a new VP framework, termed ILM-VP (iterative label mapping-based visual prompting), which automatically re-maps the source labels to the target labels and progressively improves the target task accuracy of VP. Further, when using a contrastive language-image pretrained (CLIP) model, we propose to integrate an LM process to assist the text prompt selection of CLIP and to improve the target task accuracy. Extensive experiments demonstrate that our proposal significantly outperforms state-of-the-art VP methods. As highlighted below, we show that when reprogramming an ImageNet-pretrained ResNet-18 to 13 target tasks, our method outperforms baselines by a substantial margin, e.g., 7.9% and 6.7% accuracy improvements in transfer learning to the target Flowers102 and CIFAR100 datasets. Besides, our proposal on CLIP-based VP provides 13.7% and 7.1% accuracy improvements on Flowers102 and DTD respectively. Our code is available at https://github.com/OPTML-Group/ILM-VP.

翻译：我们重新审视并推进了视觉提示（VP）——一种用于视觉任务的输入提示技术。VP通过在下游数据点中嵌入通用提示（即输入扰动模式），能够重新编程固定且预训练好的源模型，以完成目标域中的下游任务。然而，即便在源类别与目标类别之间采用无规则的标签映射（LM）时，VP为何依然有效仍是一个谜。受此启发，我们提出疑问：LM与VP之间如何相互关联？又如何利用这种关系提升目标任务的准确率？我们深入探究了LM对VP的影响，并给出肯定答案：LM的“质量”越高（通过映射精度与可解释性评估），VP的有效性越能持续提升。这与先前忽视LM因素的研究形成对比。为优化LM，我们提出了一种新型VP框架，称为ILM-VP（基于迭代标签映射的视觉提示），该框架自动将源标签重新映射至目标标签，并逐步提升VP的目标任务准确率。此外，当使用对比语言-图像预训练（CLIP）模型时，我们提议整合LM过程以辅助CLIP的文本提示选择，进而提高目标任务准确率。大量实验表明，我们的方法显著优于最先进的VP方法。亮点如下：在将ImageNet预训练的ResNet-18重新编程至13个目标任务时，我们的方法在迁移学习至目标数据集Flowers102和CIFAR100上分别取得了7.9%和6.7%的准确率提升，显著超越基线方法。同时，基于CLIP的VP方法在Flowers102和DTD上分别实现了13.7%和7.1%的准确率提升。我们的代码已开源至 https://github.com/OPTML-Group/ILM-VP。