Pre-trained language models (PLMs) have ignited a surge in demand for effective fine-tuning techniques, particularly in low-resource domains and languages. Active learning (AL), a set of algorithms designed to decrease labeling costs by minimizing label complexity, has shown promise in confronting the labeling bottleneck. In parallel, adapter modules designed for parameter-efficient fine-tuning (PEFT) have demonstrated notable potential in low-resource settings. However, the interplay between AL and adapter-based PEFT remains unexplored. We present an empirical study of PEFT behavior with AL in low-resource settings for text classification tasks. Our findings affirm the superiority of PEFT over full-fine tuning (FFT) in low-resource settings and demonstrate that this advantage persists in AL setups. We further examine the properties of PEFT and FFT through the lens of forgetting dynamics and instance-level representations, where we find that PEFT yields more stable representations of early and middle layers compared to FFT. Our research underscores the synergistic potential of AL and PEFT in low-resource settings, paving the way for advancements in efficient and effective fine-tuning.

翻译：预训练语言模型（PLMs）激起了对高效微调技术的旺盛需求，尤其在低资源领域和语言中。主动学习（AL）是一类旨在通过最小化标签复杂度来降低标注成本的算法，在应对标注瓶颈方面展现出潜力。与此同时，专为参数高效微调（PEFT）设计的适配器模块在低资源场景下已表现出显著潜力。然而，AL与基于适配器的PEFT之间的相互作用尚未得到探索。本文针对文本分类任务，在低资源场景下开展了关于PEFT结合AL行为的实证研究。我们的发现证实了PEFT在低资源场景下优于全微调（FFT），并表明这一优势在AL设置中依然保持。我们进一步通过遗忘动态和实例级表征的视角剖析了PEFT与FFT的特性，发现与FFT相比，PEFT在早期和中间层产生了更稳定的表征。本研究强调了AL与PEFT在低资源场景下的协同潜力，为推进高效且有效的微调方法铺平了道路。