Despite the demonstrated empirical efficacy of prompt tuning to adapt a pretrained language model for a new task, the theoretical underpinnings of the difference between "tuning parameters before the input" against "the tuning of model weights" are limited. We thus take one of the first steps to understand the role of soft-prompt tuning for transformer-based architectures. By considering a general purpose architecture, we analyze prompt tuning from the lens of both: universal approximation and limitations with finite-depth fixed-weight pretrained transformers for continuous-valued functions. Our universality result guarantees the existence of a strong transformer with a prompt to approximate any sequence-to-sequence function in the set of Lipschitz functions. The limitations of prompt tuning for limited-depth transformers are first proved by constructing a set of datasets, that cannot be memorized by a prompt of any length for a given single encoder layer. We also provide a lower bound on the required number of tunable prompt parameters and compare the result with the number of parameters required for a low-rank update (based on LoRA) for a single-layer setting. We finally extend our analysis to multi-layer settings by providing sufficient conditions under which the transformer can at best learn datasets from invertible functions only. Our theoretical claims are also corroborated by empirical results.

翻译：尽管提示调优在将预训练语言模型适配到新任务上展现出实证有效性，但针对"输入前参数调整"与"模型权重调优"差异的理论基础仍显不足。为此，我们率先探索了软提示调优在基于Transformer架构中的作用机理。通过考虑通用架构，我们从两个维度分析提示调优：连续值函数的普适逼近性与有限深度固定权重预训练Transformer的局限性。我们的普适性结果证明，在Lipschitz函数集中，存在具备提示功能的强Transformer可逼近任意序列到序列函数。通过构造特定数据集，我们首次证明了有限深度Transformer中提示调优的局限性——对于给定单编码器层，任何长度的提示均无法完全记忆该数据集。同时，我们推导了可调提示参数所需数量的下界，并将该结果与单层场景下基于LoRA的低秩更新所需参数数量进行了比较。最终，通过给出充分条件，我们将分析扩展至多层场景，证明Transformer最多仅能学习可逆函数构成的数据集。理论主张亦得到实证结果的验证。