While prompt tuning approaches have achieved competitive performance with high efficiency, we observe that they invariably employ the same initialization process, wherein the soft prompt is either randomly initialized or derived from an existing embedding vocabulary. In contrast to these conventional methods, this study aims to investigate an alternative way to derive soft prompt. Our empirical studies show that the soft prompt typically exhibits a low intrinsic rank characteristic. With such observations, we propose decomposed prompt tuning, a novel approach that utilizes low-rank matrices to initialize the soft prompt. Through the low-rank reparameterization, our method significantly reduces the number of trainable parameters while maintaining effectiveness. Experimental results on the SuperGLUE benchmark in both high-resource and low-resource scenarios demonstrate the effectiveness of the proposed method.

翻译：虽然提示调优方法以高效率实现了具有竞争力的性能，但我们观察到它们总是采用相同的初始化过程，其中软提示要么是随机初始化的，要么是从现有嵌入词汇表中导出的。与这些传统方法不同，本研究旨在探索一种替代方式来导出软提示。我们的实证研究表明，软提示通常表现出低本征秩特性。基于这一发现，我们提出了分解提示调优，这是一种利用低秩矩阵初始化软提示的新方法。通过低秩重参数化，我们的方法在保持有效性的同时显著减少了可训练参数的数量。在SuperGLUE基准测试上的高资源和低资源场景下的实验结果证明了所提出方法的有效性。