Prompt-Tuning is an efficient method for adapting pre-trained language models to new tasks with minimal computational overhead by modifying prompt embeddings. In this work, we investigate how crucial the phenomenon of embedding collapse, frequently observed in Prompt-Tuning, is for the final performance of the model. To address this question, we designed embedding priors and compared them with posteriors of the converged Soft and Deep Prompt-Tuning methods. Our findings suggest that priors strongly affect the position of the tuned embeddings, and models can effectively work with embeddings from different parts of activation spaces, including completely new regions. As the final Prompt-Tuning capabilities are limited, we hypothesize that controllable Prompt-Tuning posteriors may serve as a good starting point for tasks such as chain-of-thought (COT) distillation. Our experiments also show that generated trajectories are not localized in the activation space of the models. However, there are distinct clusters of activations for distant tasks (e.g., NLP and arithmetic), while activations between NLP tasks (e.g., Question-Answering and MLM) lie in the same cluster. These observations raise questions about the importance of a single activation cluster for the generalization abilities of large language models.

翻译：提示调优是一种通过修改提示嵌入来以最小计算开销将预训练语言模型适配到新任务的高效方法。在本工作中，我们研究了提示调优中经常观察到的嵌入坍缩现象对模型最终性能的关键影响。为探究此问题，我们设计了嵌入先验，并将其与收敛后的软提示调优和深度提示调优方法的后验分布进行比较。研究结果表明，先验强烈影响调优后嵌入的位置，且模型能有效处理来自激活空间不同区域（包括全新区域）的嵌入。由于最终提示调优能力存在局限，我们假设可控的提示调优后验可作为思维链蒸馏等任务的理想起点。实验还表明，生成的轨迹在模型激活空间中并非局部集中。然而，对于差异显著的任务（如自然语言处理与算术任务），其激活会形成明显聚类；而自然语言处理内部任务（如问答与掩码语言建模）的激活则处于同一聚类。这些发现引发了对单一激活聚类在大型语言模型泛化能力中重要性的思考。