Parameter-efficient tuning aims at updating only a small subset of parameters when adapting a pretrained model to downstream tasks. In this work, we introduce PASTA, in which we only modify the special token representations (e.g., [SEP] and [CLS] in BERT) before the self-attention module at each layer in Transformer-based models. PASTA achieves comparable performance to full finetuning in natural language understanding tasks including text classification and NER with up to only 0.029% of total parameters trained. Our work not only provides a simple yet effective way of parameter-efficient tuning, which has a wide range of practical applications when deploying finetuned models for multiple tasks, but also demonstrates the pivotal role of special tokens in pretrained language models

翻译：参数高效微调旨在将预训练模型适配到下游任务时仅更新少量参数。本文提出PASTA方法，该方法仅需修改Transformer模型各层自注意力模块之前的特殊标记表示（如BERT中的[SEP]和[CLS]）。在自然语言理解任务（包括文本分类和命名实体识别）中，PASTA仅需训练总参数量的0.029%即可达到与全参数微调相当的性能。本研究不仅提供了一种简单有效的参数高效微调方案，可广泛适用于多任务微调模型的实际部署场景，更揭示了特殊标记在预训练语言模型中的关键作用。