Visual Prompt Tuning (VPT) has proven effective for parameter-efficient adaptation of pre-trained vision models to downstream tasks by inserting task-specific learnable prompt tokens. Despite its empirical success, a comprehensive theoretical understanding of VPT remains an active area of research. Building on the recently established connection between Mixture of Experts (MoE) and prompt-based methods, wherein each attention head can be conceptualized as a composition of multiple MoE models, we reinterpret VPT as the introduction of new prompt experts into these MoE structures. We identify a key limitation in existing VPT frameworks: the restricted functional expressiveness of prompt experts, which remain static and thus limited in their adaptability. To address this, we propose Visual Adaptive Prompt Tuning (VAPT), a novel method that endows prompt experts with enhanced expressiveness while preserving parameter efficiency. Empirical evaluations on VTAB-1K and FGVC demonstrate that VAPT achieves substantial performance improvements, surpassing fully fine-tuned baselines by 7.34% and 1.04%, respectively. Moreover, VAPT consistently outperforms VPT while requiring fewer additional parameters. Furthermore, our theoretical analysis indicates that VAPT achieves optimal sample efficiency. Collectively, these results underscore the theoretical grounding and empirical advantages of our approach.

翻译：视觉提示调优（VPT）通过在预训练视觉模型中插入任务特定的可学习提示令牌，已被证明能有效实现参数高效的下游任务适应。尽管其经验上取得了成功，但对VPT的全面理论理解仍是活跃的研究领域。基于最近建立的混合专家（MoE）与基于提示的方法之间的联系——其中每个注意力头可被概念化为多个MoE模型的组合——我们将VPT重新解释为向这些MoE结构中引入新的提示专家。我们指出了现有VPT框架的一个关键局限：提示专家的功能表达能力受限，它们保持静态，因此在适应性方面存在限制。为解决这一问题，我们提出了视觉自适应提示调优（VAPT），这是一种新颖的方法，在保持参数效率的同时，赋予提示专家更强的表达能力。在VTAB-1K和FGVC上的实证评估表明，VAPT实现了显著的性能提升，分别超过完全微调基线7.34%和1.04%。此外，VAPT在需要更少额外参数的情况下，始终优于VPT。进一步的理论分析表明，VAPT实现了最优的样本效率。综合来看，这些结果凸显了我们方法的理论基础和实证优势。