With the advance of large-scale model technologies, parameter-efficient transfer learning (PETL) has swept across various fields of Artificial Intelligence. Its core idea is to adapt the model to downstream tasks using only a small number of parameters. Recently, some studies have applied these techniques proven effective to multimodal tasks. However, two critical issues remain unresolved: how to further reduce the complexity with lightweight design and how to boost alignment between modalities under extremely low parameters. In this paper, we propose A graceful prompt framework for cross-modal transfer (Aurora) to overcome these challenges. Considering the redundancy in existing architectures, we first utilize the mode approximation to generate few trainable parameters to implement the multi-modal prompt tuning, which explores the low intrinsic dimension with only 0.05% parameters of the pre-trained model. Then, to better narrow the modality gap, we propose the informative context enhancement and gated query transformation modules under extremely few parameters scenes. A thorough evaluation of the Aurora on six cross-modal downstream benchmarks shows that it not only outperforms the state-of-the-art, but even outperforms the full fine-tuning approach. Our code is available at: https://github.com/WillDreamer/Aurora.

翻译：随着大规模模型技术的发展，参数高效迁移学习（PETL）已席卷人工智能各领域。其核心思想是仅使用少量参数即可使模型适应下游任务。近年来，部分研究将这些被证明有效的技术应用于多模态任务。然而，仍有两个关键问题尚未解决：如何通过轻量化设计进一步降低复杂度，以及如何在极低参数条件下增强模态间的对齐能力。本文提出一种优雅的跨模态迁移提示框架（Aurora）以应对这些挑战。鉴于现有架构中的冗余性，我们首先利用模态近似生成少量可训练参数实现多模态提示调优，该方法仅需预训练模型0.05%的参数即可探索低本征维度。随后，为更好地缩小模态差距，我们提出在极低参数场景下的信息上下文增强与门控查询变换模块。在六个跨模态下游基准任务上对Aurora的全面评估表明，该框架不仅超越当前最先进方法，甚至优于全参数微调方法。我们的代码已开源在：https://github.com/WillDreamer/Aurora。