Large Language Models (LLMs), with billions of parameters, present significant challenges for full finetuning due to the high computational demands, memory requirements, and impracticality of many real-world applications. When faced with limited computational resources or small datasets, updating all model parameters can often result in overfitting. To address this, lightweight finetuning techniques have been proposed, like learning low-rank adapter layers. These methods aim to train only a few additional parameters combined with the base model, which remains frozen, reducing resource usage and mitigating overfitting risks. In this work, we propose an adaptor model based on stochastic gates that simultaneously sparsify the frozen base model with task-specific adaptation. Our method comes with a small number of trainable parameters and allows us to speed up the base model inference with competitive accuracy. We evaluate it in additional variants by equipping it with additional low-rank parameters and comparing it to several recent baselines. Our results show that the proposed method improves the finetuned model accuracy comparatively to the several baselines and allows the removal of up to 20-40\% without significant accuracy loss.

翻译：大型语言模型（LLMs）具有数十亿参数，由于计算需求高、内存占用大，且在许多实际应用中难以部署，其全参数微调面临重大挑战。在计算资源有限或数据集规模较小时，更新全部模型参数往往会导致过拟合。为此，研究者提出了轻量级微调技术，例如学习低秩适配器层。这类方法仅训练少量附加参数并与冻结的基模型结合，从而降低资源消耗并缓解过拟合风险。本文提出一种基于随机门的适配器模型，该模型能在进行任务特定适配的同时对冻结的基模型进行稀疏化处理。该方法仅需训练少量参数，可在保持竞争力的精度前提下加速基模型推理。我们通过为其配备额外低秩参数构建多种变体，并与若干近期基线方法进行比较评估。实验结果表明，相较于多种基线方法，所提方案能相对提升微调模型的精度，并允许移除20-40%的参数而不造成显著精度损失。