Prompt tuning has become a popular strategy for adapting Vision-Language Models (VLMs) to zero/few-shot visual recognition tasks. Some prompting techniques introduce prior knowledge due to its richness, but when learnable tokens are randomly initialized and disconnected from prior knowledge, they tend to overfit on seen classes and struggle with domain shifts for unseen ones. To address this issue, we propose the InPK model, which infuses class-specific prior knowledge into the learnable tokens during initialization, thus enabling the model to explicitly focus on class-relevant information. Furthermore, to mitigate the weakening of class information by multi-layer encoders, we continuously reinforce the interaction between learnable tokens and prior knowledge across multiple feature levels. This progressive interaction allows the learnable tokens to better capture the fine-grained differences and universal visual concepts within prior knowledge, enabling the model to extract more discriminative and generalized text features. Even for unseen classes, the learned interaction allows the model to capture their common representations and infer their appropriate positions within the existing semantic structure. Moreover, we introduce a learnable text-to-vision projection layer to accommodate the text adjustments, ensuring better alignment of visual-text semantics. Extensive experiments on 11 recognition datasets show that InPK significantly outperforms state-of-the-art methods in multiple zero/few-shot image classification tasks.

翻译：提示调优已成为将视觉语言模型（VLMs）适应于零样本/少样本视觉识别任务的流行策略。一些提示技术因其丰富性而引入先验知识，但当可学习标记随机初始化且与先验知识脱节时，它们容易对已见类别过拟合，并在面对未见类别的领域偏移时表现不佳。为解决这一问题，我们提出了InPK模型，该模型在初始化阶段将类别特定的先验知识注入可学习标记中，从而使模型能够显式关注与类别相关的信息。此外，为减轻多层编码器对类别信息的弱化，我们在多个特征层级持续强化可学习标记与先验知识之间的交互。这种渐进式交互使可学习标记能更好地捕捉先验知识中的细粒度差异和通用视觉概念，从而使模型能够提取更具判别性和泛化性的文本特征。即使对于未见类别，学习到的交互也使模型能够捕捉其共同表征，并推断其在现有语义结构中的适当位置。此外，我们引入了一个可学习的文本到视觉投影层，以适应文本调整，确保视觉-文本语义更好地对齐。在11个识别数据集上的大量实验表明，InPK在多个零样本/少样本图像分类任务中显著优于现有最先进方法。