As Multimodal Large Language Models (MLLMs) grow in size, adapting them to specialized tasks becomes increasingly challenging due to high computational and memory demands. Indeed, traditional fine-tuning methods are costly, due to the need for extensive, task-specific training. While efficient adaptation methods exist that aim to reduce these costs, in practice they suffer from shallow inter-modal alignment, which severely hurts model effectiveness. To tackle these computational challenges and improve inter-modal alignment, we introduce the MultiWay-Adapter (MWA), a novel framework featuring an 'Alignment Enhancer'. This enhancer deepens inter-modal alignment, enabling high transferability with minimal tuning effort. Our experiments show that unlike prior efficient tuning approaches, MWA maintains model effectiveness, while reducing training time by up-to 57%. MWA is also lightweight, increasing model size by only 2-3% (in terms of parameters) for state-of-the-art foundation models like BEiT-3 Large. These results demonstrate that MWA provides an efficient and effective adaptation method for MLLMs, significantly broadening their applicability.

翻译：摘要：随着多模态大语言模型（MLLMs）规模不断增长，由于高昂的计算和内存需求，将其适配至特定任务变得日益困难。传统微调方法因需要大量任务特定训练而成本高昂。尽管存在旨在降低成本的效率适配方法，但实践中这些方法存在模态间对齐深度不足的问题，严重损害模型有效性。为应对计算挑战并增强模态间对齐，我们提出MultiWay-Adapter（MWA）这一新型框架，其核心是"对齐增强器"（Alignment Enhancer）。该增强器深化了模态间对齐，从而以最小调优代价实现高迁移性。实验表明，与现有高效调优方法不同，MWA在保持模型有效性的同时可将训练时间缩减高达57%。MWA还具备轻量化特性，仅以2-3%的参数增幅即可适配BEiT-3 Large等现有基础模型。这些结果证明，MWA为MLLMs提供了高效且有效的适配方法，显著拓展了其应用范围。