Large language models achieve state-of-the-art performance but are increasingly costly to fine-tune. Prompt tuning is a parameter-efficient fine-tuning method that addresses parameter-efficiency by learning prompt embeddings, but these embeddings are typically tied to the model's hidden dimensionality, limiting parameter saving. In this paper, we propose Ultra-Low-dimensional Prompt Tuning (ULPT), a simple yet effective method that optimizes prompts in a low-dimensional space (e.g., 2D) and uses a frozen random matrix for up-projection. ULPT can achieve 98% reduction in the training parameters compared to vanilla prompt tuning while preserving performance. Our extensive experiments across over 20 NLP tasks demonstrate that ULPT consistently outperforms recent parameter-efficient tuning methods using significantly fewer parameters, making it well-suited as a storage-efficient framework for massive LLM customization.

翻译：大型语言模型虽能实现最先进的性能，但其微调成本日益高昂。提示调优是一种参数高效的微调方法，通过学习提示嵌入来解决参数效率问题，然而这些嵌入通常与模型的隐藏维度绑定，限制了参数节省。本文提出超低维提示调优（ULPT），这是一种简单而有效的方法，其在低维空间（例如2维）中优化提示，并使用固定的随机矩阵进行上投影。与原始提示调优相比，ULPT能在保持性能的同时实现98%的训练参数削减。我们在超过20项自然语言处理任务上的广泛实验表明，ULPT在使用显著更少参数的情况下，始终优于近期提出的参数高效调优方法，使其成为适用于大规模LLM定制的高存储效率框架。