As the size of the pre-trained language model (PLM) continues to increase, numerous parameter-efficient transfer learning methods have been proposed recently to compensate for the tremendous cost of fine-tuning. Despite the impressive results achieved by large pre-trained language models (PLMs) and various parameter-efficient transfer learning (PETL) methods on sundry benchmarks, it remains unclear if they can handle inputs that have been distributionally shifted effectively. In this study, we systematically explore how the ability to detect out-of-distribution (OOD) changes as the size of the PLM grows or the transfer methods are altered. Specifically, we evaluated various PETL techniques, including fine-tuning, Adapter, LoRA, and prefix-tuning, on three different intention classification tasks, each utilizing various language models with different scales.

翻译：随着预训练语言模型（PLM）规模的持续增长，近年来涌现出大量参数高效的迁移学习方法，以弥补微调带来的巨大成本。尽管大型预训练语言模型（PLM）及各种参数高效迁移学习（PETL）方法在各类基准测试中取得了显著成果，但其能否有效处理经过分布偏移的输入仍不明确。本研究系统探索了当PLM规模扩大或迁移方法改变时，模型检测分布外（OOD）样本的能力如何变化。具体而言，我们在三项不同的意图分类任务中评估了多种PETL技术（包括微调、适配器、LoRA和前缀微调），每项任务均使用不同规模的多种语言模型进行实验。