Prompt learning has achieved great success in efficiently exploiting large-scale pre-trained models in natural language processing (NLP). It reformulates the downstream tasks as the generative pre-training ones to achieve consistency, thus improving the performance stably. However, when transferring it to the vision area, current visual prompt learning methods are almost designed on discriminative pre-trained models, and there is also a lack of careful design to unify the forms of pre-training and downstream tasks. To explore prompt learning on the generative pre-trained visual model, as well as keeping the task consistency, we propose Visual Prompt learning as masked visual Token Modeling (VPTM) to transform the downstream visual classification into the pre-trained masked visual token prediction. In addition, we develop the prototypical verbalizer for mapping the predicted visual token with implicit semantics to explicit downstream labels. To our best knowledge, VPTM is the first visual prompt method on the generative pre-trained visual model, which achieves consistency between pre-training and downstream visual classification by task reformulation. Experiments show that VPTM outperforms other visual prompt methods and achieves excellent efficiency. Moreover, the task consistency of VPTM contributes to the robustness against prompt location, prompt length and prototype dimension, and could be deployed uniformly.

翻译：提示学习在自然语言处理（NLP）中高效利用大规模预训练模型方面取得了巨大成功。它通过将下游任务重构为生成式预训练任务以实现一致性，从而稳定提升性能。然而，当将其迁移至视觉领域时，当前的视觉提示学习方法几乎都基于判别式预训练模型设计，且缺乏统一预训练与下游任务形式的精细设计。为了在生成式预训练视觉模型上探索提示学习，并保持任务一致性，我们提出将视觉提示学习建模为掩码视觉令牌预测（VPTM），从而将下游视觉分类任务转化为预训练掩码视觉令牌预测任务。此外，我们开发了原型化语言表达器，用于将具有隐式语义的预测视觉令牌映射为显式下游标签。据我们所知，VPTM是首个基于生成式预训练视觉模型的视觉提示方法，通过任务重构实现了预训练与下游视觉分类的一致性。实验表明，VPTM优于其他视觉提示方法，并实现了卓越的效率。此外，VPTM的任务一致性使其对提示位置、提示长度和原型维度具有鲁棒性，并能够进行统一部署。