Fine-tuning pre-trained transformer models, e.g., Swin Transformer, are successful in numerous downstream for dense prediction vision tasks. However, one major issue is the cost/storage of their huge amount of parameters, which becomes increasingly challenging to handle with the growing amount of vision tasks. In this paper, we propose an effective approach to alleviate the issue, namely selective feature adapter (SFA). It achieves state-of-the-art (SoTA) performance under any given budget of trainable parameters, and demonstrates comparable or better performance than fully fine-tuned models across various dense tasks. Specifically, SFA consists of external adapters and internal adapters which are sequentially operated over a transformer model. For external adapters, we properly select the places and amount of additional multilayer perception (MLP). For internal adapters, we transform a few task-important parameters inside the transformer, which are automatically discovered through a simple yet effective lottery ticket algorithm. Our experiments show that the dual adapter module, a.k.a SFA, is essential to achieve the best trade-off on dense vision tasks, such as segmentation, detection and depth-estimation, outperforming other adapters with a single module.

翻译：微调预训练的Transformer模型（例如Swin Transformer）在众多密集预测视觉任务的下游应用中取得了成功。然而，一个主要问题是其庞大参数规模带来的成本/存储挑战，随着视觉任务数量的增长，这一问题愈发棘手。本文提出了一种有效缓解该问题的方法，即选择性特征适配器（SFA）。它能在任何给定的可训练参数预算下实现最先进的性能，并在各类密集任务中展现出与完全微调模型相当或更优的表现。具体而言，SFA由外部适配器和内部适配器组成，两者按顺序在Transformer模型上运作。对于外部适配器，我们合理选择额外多层感知机（MLP）的添加位置和数量。对于内部适配器，我们转换Transformer内部少量任务关键参数，这些参数通过一种简单而有效的彩票票算法自动发现。实验表明，双适配器模块（即SFA）对于在分割、检测和深度估计等密集视觉任务中实现最佳权衡至关重要，其性能优于仅含单一模块的其他适配器。