Adapters are widely popular parameter-efficient transfer learning approaches in natural language processing that insert trainable modules in between layers of a pre-trained language model. Apart from several heuristics, however, there has been a lack of studies analyzing the optimal number of adapter parameters needed for downstream applications. In this paper, we propose an adapter pruning approach by studying the tropical characteristics of trainable modules. We cast it as an optimization problem that aims to prune parameters from the adapter layers without changing the orientation of underlying tropical hypersurfaces. Our experiments on five NLP datasets show that tropical geometry tends to identify more relevant parameters to prune when compared with the magnitude-based baseline, while a combined approach works best across the tasks.

翻译：适配器是自然语言处理中广泛流行的参数高效迁移学习方法，它通过在预训练语言模型的层间插入可训练模块来实现。然而，除了一些启发式方法外，目前缺乏对下游应用所需适配器参数最优数量的分析研究。本文提出了一种通过研究可训练模块的热带特征进行适配器剪枝的方法。我们将该问题建模为一个优化问题，旨在剪枝适配器层中的参数，同时保持底层热带超曲面的方向不变。我们在五个自然语言处理数据集上的实验表明，与基于幅度的基线方法相比，热带几何倾向于识别更相关的剪枝参数，而联合方法在各任务中表现最佳。