As the size of the pre-trained language model (PLM) continues to increase, numerous parameter-efficient transfer learning methods have been proposed recently to compensate for the tremendous cost of fine-tuning. Despite the impressive results achieved by large pre-trained language models (PLMs) and various parameter-efficient transfer learning (PETL) methods on sundry benchmarks, it remains unclear if they can handle inputs that have been distributionally shifted effectively. In this study, we systematically explore how the ability to detect out-of-distribution (OOD) changes as the size of the PLM grows or the transfer methods are altered. Specifically, we evaluated various PETL techniques, including fine-tuning, Adapter, LoRA, and prefix-tuning, on three different intention classification tasks, each utilizing various language models with different scales.

翻译：随着预训练语言模型（PLM）规模持续增大，近年来涌现出大量参数高效迁移学习方法以弥补微调的巨大成本。尽管大型预训练语言模型（PLM）及各类参数高效迁移学习（PETL）方法在各种基准测试中取得了令人瞩目的成果，但尚不清楚它们能否有效处理经过分布偏移的输入。本研究系统探究了当PLM规模扩大或迁移方法改变时，检测分布外（OOD）样本能力的变化规律。具体而言，我们在三个不同意图分类任务中评估了包括微调、Adapter、LoRA和前缀微调在内的多种PETL技术，每个任务均采用不同规模的语言模型进行测试。