Syntactically controlled paraphrase generation requires language models to generate paraphrases for sentences according to specific syntactic structures. Existing fine-tuning methods for this task are costly as all the parameters of the model need to be updated during the training process. Inspired by recent studies on parameter-efficient learning, we propose Parse-Instructed Prefix (PIP), a novel adaptation of prefix-tuning to tune large pre-trained language models on syntactically controlled paraphrase generation task in a low-data setting with significantly less training cost. We introduce two methods to instruct a model's encoder prefix to capture syntax-related knowledge: direct initiation (PIP-Direct) and indirect optimization (PIP-Indirect). In contrast to traditional fine-tuning methods for this task, PIP is a compute-efficient alternative with 10 times less learnable parameters. Compared to existing prefix-tuning methods, PIP excels at capturing syntax control information, achieving significantly higher performance at the same level of learnable parameter count.

翻译：摘要：句法受控释义生成要求语言模型根据特定句法结构为句子生成释义。现有针对该任务的微调方法成本高昂，因为训练过程中需要更新模型所有参数。受近期参数高效学习研究的启发，我们提出解析指导前缀（Parse-Instructed Prefix, PIP）——一种对前缀微调（prefix-tuning）的创新性改进，旨在以显著降低的训练成本，在低数据场景下对大型预训练语言模型进行句法受控释义生成任务调优。我们引入两种方法来指导模型编码器前缀捕获句法相关知识：直接初始化法（PIP-Direct）和间接优化法（PIP-Indirect）。与传统针对该任务的微调方法相比，PIP是一种计算高效的替代方案，可学习参数减少10倍；与现有前缀微调方法相比，PIP在捕获句法控制信息方面表现更优，在同等可学习参数规模下实现了显著更高的性能。