As the size of the pre-trained language model (PLM) continues to increase, numerous parameter-efficient transfer learning methods have been proposed recently to compensate for the tremendous cost of fine-tuning. Despite the impressive results achieved by large pre-trained language models (PLMs) and various parameter-efficient transfer learning (PETL) methods on sundry benchmarks, it remains unclear if they can handle inputs that have been distributionally shifted effectively. In this study, we systematically explore how the ability to detect out-of-distribution (OOD) changes as the size of the PLM grows or the transfer methods are altered. Specifically, we evaluated various PETL techniques, including fine-tuning, Adapter, LoRA, and prefix-tuning, on three different intention classification tasks, each utilizing various language models with different scales.

翻译：随着预训练语言模型（PLM）规模持续增大，近年来提出了众多参数高效迁移学习方法以弥补微调带来的巨大成本。尽管大型预训练语言模型（PLM）和各类参数高效迁移学习（PETL）方法在多个基准测试中取得了令人瞩目的成果，但目前尚不清楚它们能否有效处理经历分布偏移的输入。本研究系统探索了当PLM规模扩大或迁移方法改变时，分布外（OOD）检测能力的变化规律。具体而言，我们在三项不同的意图分类任务中评估了包括微调、Adapter、LoRA和前缀微调（prefix-tuning）在内的多种PETL技术，每项任务均使用了不同规模的语言模型。