The Outstanding performance and growing size of Large Language Models has led to increased attention in parameter efficient learning. The two predominant approaches are Adapters and Pruning. Adapters are to freeze the model and give it a new weight matrix on the side, which can significantly reduce the time and memory of training, but the cost is that the evaluation and testing will increase the time and memory consumption. Pruning is to cut off some weight and re-distribute the remaining weight, which sacrifices the complexity of training at the cost of extremely high memory and training time, making the cost of evaluation and testing relatively low. So efficiency of training and inference can't be obtained in the same time. In this work, we propose a task-oriented Pruning-Adapter method that achieve a high memory efficiency of training and memory, and speeds up training time and ensures no significant decrease in accuracy in GLUE tasks, achieving training and inference efficiency at the same time.

翻译：大型语言模型卓越的性能和不断增长的规模使得参数高效学习引起了更多关注。两种主要方法分别是适配器（Adapter）与剪枝（Pruning）。适配器通过冻结模型并在其旁边添加新的权重矩阵，显著降低了训练的时间和内存消耗，但代价是评估和测试阶段会增加时间和内存开销。剪枝则是切除部分权重并重新分配剩余权重，牺牲了训练的复杂度，以极高的内存和训练时间为代价，使得评估和测试阶段的成本相对较低。因此，训练和推理的效率无法同时获得。在这项工作中，我们提出了一种面向任务的剪枝-适配器方法，该方法在GLUE任务上实现了训练和内存的高效性，同时加快了训练速度并确保准确率没有显著下降，从而同时实现了训练和推理的高效率。