We present Diversity-Aware Meta Visual Prompting~(DAM-VP), an efficient and effective prompting method for transferring pre-trained models to downstream tasks with frozen backbone. A challenging issue in visual prompting is that image datasets sometimes have a large data diversity whereas a per-dataset generic prompt can hardly handle the complex distribution shift toward the original pretraining data distribution properly. To address this issue, we propose a dataset Diversity-Aware prompting strategy whose initialization is realized by a Meta-prompt. Specifically, we cluster the downstream dataset into small homogeneity subsets in a diversity-adaptive way, with each subset has its own prompt optimized separately. Such a divide-and-conquer design reduces the optimization difficulty greatly and significantly boosts the prompting performance. Furthermore, all the prompts are initialized with a meta-prompt, which is learned across several datasets. It is a bootstrapped paradigm, with the key observation that the prompting knowledge learned from previous datasets could help the prompt to converge faster and perform better on a new dataset. During inference, we dynamically select a proper prompt for each input, based on the feature distance between the input and each subset. Through extensive experiments, our DAM-VP demonstrates superior efficiency and effectiveness, clearly surpassing previous prompting methods in a series of downstream datasets for different pretraining models. Our code is available at: \url{https://github.com/shikiw/DAM-VP}.

翻译：我们提出多样性感知的元视觉提示方法（Diversity-Aware Meta Visual Prompting，DAM-VP），这是一种高效且有效的提示方法，用于将预训练模型迁移至下游任务，并保持骨干网络冻结。视觉提示中的一个挑战性问题在于，图像数据集有时存在较大的数据多样性，而针对每个数据集的通用提示难以恰当处理相对于原始预训练数据分布的复杂分布偏移。为解决这一问题，我们提出一种数据集多样性感知的提示策略，其初始化通过元提示（Meta-prompt）实现。具体而言，我们以多样性自适应方式将下游数据集聚类为若干同质子集，每个子集分别优化其专属提示。这种分而治之的设计大幅降低了优化难度，并显著提升了提示性能。此外，所有提示均通过元提示进行初始化，该元提示在多个数据集上学习得到。这是一种自举式范式，其关键观察在于从先前数据集学到的提示知识能够帮助新数据集上的提示更快收敛并获得更优性能。在推理阶段，我们根据输入与每个子集的特征距离，动态为每个输入选择合适的提示。通过大量实验，我们的DAM-VP展现了卓越的效率和效果，在多个下游数据集上针对不同预训练模型均明显优于先前的提示方法。我们的代码开源于：\url{https://github.com/shikiw/DAM-VP}。