Recently, transformers have shown strong ability as visual feature extractors, surpassing traditional convolution-based models in various scenarios. However, the success of vision transformers largely owes to their capacity to accommodate numerous parameters. As a result, new challenges for adapting large models to downstream tasks arise. On the one hand, classic fine-tuning tunes all parameters in a huge model for every task and thus easily falls into overfitting, leading to inferior performance. On the other hand, on resource-limited devices, fine-tuning stores a full copy of parameters and thus is usually impracticable for the shortage of storage space. However, few works have focused on how to efficiently and effectively transfer knowledge in a vision transformer. Existing methods did not dive into the properties of visual features, leading to inferior performance. Moreover, some of them bring heavy inference cost though benefiting storage. To tackle these problems, we propose consolidator to modify the pre-trained model with the addition of a small set of tunable parameters to temporarily store the task-specific knowledge while freezing the backbone model. Motivated by the success of group-wise convolution, we adopt grouped connections across the features extracted by fully connected layers to construct tunable parts in a consolidator. To further enhance the model's capacity to transfer knowledge under a constrained storage budget and keep inference efficient, we consolidate the parameters in two stages: 1. between adaptation and storage, and 2. between loading and inference. On a series of downstream visual tasks, our consolidator can reach up to 7.56 better accuracy than full fine-tuning with merely 0.35% parameters, and outperform state-of-the-art parameter-efficient tuning methods by a clear margin. Code is available at https://github.com/beyondhtx/Consolidator.

翻译：近期，Transformer在视觉特征提取方面展现出强大能力，已在多种场景中超越传统基于卷积的模型。然而，视觉Transformer的成功很大程度上归功于其容纳大量参数的能力，这为大型模型在下游任务中的适配带来了新挑战。一方面，经典微调方法需为每个任务调整完整模型的全部参数，易导致过拟合而性能欠佳；另一方面，在资源受限设备上，微调需存储完整参数副本，常因存储空间不足而难以实现。目前鲜有研究聚焦于如何高效有效地迁移视觉Transformer中的知识，现有方法未深入探究视觉特征的特性因而性能欠佳，部分方法虽节省存储却带来高昂推理成本。为解决这些问题，我们提出Consolidator，通过在冻结主干模型的同时添加少量可调参数，临时存储任务特定知识来修正预训练模型。受分组卷积成功经验的启发，我们在全连接层提取的特征间采用分组连接来构建Consolidator中的可调部分。为在有限存储预算下增强模型知识迁移能力并保持推理效率，我们在两个阶段进行参数整合：1）适配与存储之间；2）加载与推理之间。在一系列下游视觉任务中，我们的Consolidator仅用0.35%的参数即可达到比全微调高出7.56个百分点的准确率，并以显著优势超越当前最优的参数高效微调方法。代码已开源：https://github.com/beyondhtx/Consolidator。