In the arena of language model fine-tuning, the traditional approaches, such as Domain-Adaptive Pretraining (DAPT) and Task-Adaptive Pretraining (TAPT), although effective, but computational intensive. This research introduces a novel adaptation method utilizing the UniPELT framework as a base and added a PromptTuning Layer, which significantly reduces the number of trainable parameters while maintaining competitive performance across various benchmarks. Our method employs adapters, which enable efficient transfer of pretrained models to new tasks with minimal retraining of the base model parameters. We evaluate our approach using three diverse datasets: the GLUE benchmark, a domain-specific dataset comprising four distinct areas, and the Stanford Question Answering Dataset 1.1 (SQuAD). Our results demonstrate that our customized adapter-based method achieves performance comparable to full model fine-tuning, DAPT+TAPT and UniPELT strategies while requiring fewer or equivalent amount of parameters. This parameter efficiency not only alleviates the computational burden but also expedites the adaptation process. The study underlines the potential of adapters in achieving high performance with significantly reduced resource consumption, suggesting a promising direction for future research in parameter-efficient fine-tuning.

翻译：在语言模型微调领域中，传统方法如领域自适应预训练（DAPT）和任务自适应预训练（TAPT）虽有效但计算成本高昂。本研究提出一种新型自适应方法，以UniPELT框架为基础并引入提示微调层（PromptTuning Layer），在显著减少可训练参数数量的同时，保持各基准测试中的竞争性性能。我们的方法采用适配器，使预训练模型能以最小代价重新训练基础模型参数，高效迁移至新任务。我们使用三个多样化数据集评估该方法：GLUE基准测试、涵盖四个领域的专用数据集以及斯坦福问答数据集1.1（SQuAD）。结果表明，与完整模型微调、DAPT+TAPT及UniPELT策略相比，我们定制的适配器方法在参数数量更少或相当的情况下实现了可比的性能。这种参数效率不仅减轻了计算负担，还加速了自适应过程。本研究凸显了适配器在显著降低资源消耗的同时保持高性能的潜力，为参数高效微调的未来研究指明了方向。