Visual Prompt Tuning (VPT) adapts a frozen Vision Transformer (ViT) to downstream tasks by inserting a small number of learnable prompt tokens into the token sequence at each layer. However, we observe that existing VPT variants often suffer from unstable training dynamics, characterized by gradient oscillations. A layer-wise analysis reveals that shallow-layer prompts tend to stagnate early, while deeper-layer prompts exhibit high-variance oscillations, leading to cross-layer mismatch. These issues slow convergence and degrade final performance. To address these challenges, we propose Prompt-Agnostic Evolution ($\mathtt{PAE}$), which strengthens vision prompt tuning by explicitly modeling prompt dynamics. From a frequency-domain perspective, we initialize prompts in a task-aware direction by uncovering and propagating frequency shortcut patterns that the backbone inherently exploits for recognition. To ensure coherent evolution across layers, we employ a shared Koopman operator that imposes a global linear transformation instead of uncoordinated, layer-specific updates. Finally, inspired by Lyapunov stability theory, we introduce a regularizer that constrains error amplification during evolution. Extensive experiments show that $\mathtt{PAE}$ accelerates convergence with an average $1.41\times$ speedup and improves accuracy by 1-3% on 25 datasets across multiple downstream tasks. Beyond performance, $\mathtt{PAE}$ is prompt-agnostic and lightweight, and it integrates seamlessly with diverse VPT variants without backbone modification or inference-time changes.

翻译：视觉提示调优（VPT）通过在每个层的令牌序列中插入少量可学习的提示令牌，使冻结的视觉Transformer（ViT）适应下游任务。然而，我们观察到现有的VPT变体常常遭受训练动态不稳定的困扰，其特点是梯度振荡。逐层分析表明，浅层提示往往过早停滞，而深层提示则表现出高方差振荡，导致跨层不匹配。这些问题减缓了收敛速度并降低了最终性能。为应对这些挑战，我们提出了提示无关演化（$\mathtt{PAE}$），它通过显式建模提示动态来增强视觉提示调优。从频域视角出发，我们通过揭示并传播主干网络在识别过程中固有利用的频率捷径模式，将提示初始化在任务感知的方向上。为确保跨层的连贯演化，我们采用共享的Koopman算子，施加全局线性变换，而非不协调的、层特定的更新。最后，受李雅普诺夫稳定性理论启发，我们引入了一个正则化器，以约束演化过程中的误差放大。大量实验表明，$\mathtt{PAE}$在多个下游任务的25个数据集上，平均加速收敛$1.41\times$，并将准确率提高1-3%。除了性能提升，$\mathtt{PAE}$具有提示无关性和轻量级特性，并且无需修改主干网络或改变推理过程，即可与多种VPT变体无缝集成。