With ever increasing parameters and computation, vision-language pre-trained (VLP) models exhibit prohibitive expenditure in downstream task adaption. Recent endeavors mainly focus on parameter efficient transfer learning (PETL) for VLP models by only updating a small number of parameters. However, excessive computational overhead still plagues the application of VLPs. In this paper, we aim at parameter and computation efficient transfer learning (PCETL) for VLP models. In particular, PCETL not only needs to limit the number of trainable parameters in VLP models, but also to reduce the computational redundancy during inference, thus enabling a more efficient transfer. To approach this target, we propose a novel dynamic architecture skipping (DAS) approach towards effective PCETL. Instead of directly optimizing the intrinsic architectures of VLP models, DAS first observes the significances of their modules to downstream tasks via a reinforcement learning (RL) based process, and then skips the redundant ones with lightweight networks, i.e., adapters, according to the obtained rewards. In this case, the VLP model can well maintain the scale of trainable parameters while speeding up its inference on downstream tasks. To validate DAS, we apply it to two representative VLP models, namely ViLT and METER, and conduct extensive experiments on a bunch of VL tasks. The experimental results not only show the great advantages of DAS in reducing computational complexity, e.g. -11.97% FLOPs of METER on VQA2.0, but also confirm its competitiveness against existing PETL methods in terms of parameter scale and performance. Our source code is given in our appendix.

翻译：随着参数规模和计算量的持续增长，视觉-语言预训练（VLP）模型在下游任务适配中展现出高昂的计算成本。近期研究主要聚焦于VLP模型的参数高效迁移学习（PETL），仅更新少量参数。然而，过度的计算开销仍制约着VLP模型的应用。本文针对VLP模型提出参数与计算高效迁移学习（PCETL）方法。具体而言，PCETL不仅需限制VLP模型的可训练参数数量，还需减少推理过程中的计算冗余，从而实现更高效的迁移。为此，我们提出一种新颖的动态架构跳跃（DAS）方法以实现有效的PCETL。DAS不直接优化VLP模型的内在架构，而是首先通过基于强化学习（RL）的过程观测各模块对下游任务的重要性，随后根据获得的奖励信号，使用轻量级网络（即适配器）跳过冗余模块。在此机制下，VLP模型既能保持可训练参数规模，又可加速下游任务的推理过程。为验证DAS的有效性，我们将其应用于两个代表性VLP模型——ViLT和METER，并在多个视觉-语言任务上开展广泛实验。实验结果不仅表明DAS在降低计算复杂度方面具有显著优势（例如在VQA2.0任务上使METER的FLOPs减少11.97%），还证实其在参数规模与性能方面相比现有PETL方法的竞争力。相关源代码详见附录。