Large Language Models (LLMs) have so far impressed the world, with unprecedented capabilities that emerge in models at large scales. On the vision side, transformer models (i.e., ViT) are following the same trend, achieving the best performance on challenging benchmarks. With the abundance of such unimodal models, a natural question arises; do we need also to follow this trend to tackle multimodal tasks? In this work, we propose to rather direct effort to efficient adaptations of existing models, and propose to augment Language Models with perception. Existing approaches for adapting pretrained models for vision-language tasks still rely on several key components that hinder their efficiency. In particular, they still train a large number of parameters, rely on large multimodal pretraining, use encoders (e.g., CLIP) trained on huge image-text datasets, and add significant inference overhead. In addition, most of these approaches have focused on Zero-Shot and In Context Learning, with little to no effort on direct finetuning. We investigate the minimal computational effort needed to adapt unimodal models for multimodal tasks and propose a new challenging setup, alongside different approaches, that efficiently adapts unimodal pretrained models. We show that by freezing more than 99% of total parameters, training only one linear projection layer, and prepending only one trainable token, our approach (dubbed eP-ALM) significantly outperforms other baselines on VQA and Captioning across Image, Video, and Audio modalities, following the proposed setup. The code is available here: https://github.com/mshukor/eP-ALM.

翻译：大型语言模型（LLMs）以其在大规模模型中涌现的前所未有能力，至今令世界瞩目。在视觉领域，Transformer模型（如ViT）也遵循相同趋势，在挑战性基准测试中取得最优性能。面对此类单模态模型的丰富性，一个自然的问题随之产生：我们是否也需要遵循这一趋势来处理多模态任务？本文提出将精力转向对现有模型进行高效适配，并建议对语言模型进行感知增强。现有面向视觉语言任务的预训练模型适配方法，仍依赖若干阻碍其效率的关键组件。具体而言，它们仍需训练大量参数、依赖大规模多模态预训练、使用在巨量图文数据集上训练的编码器（如CLIP），并引入显著的推理开销。此外，多数方法聚焦于零样本（Zero-Shot）和上下文学习（In Context Learning），对直接微调几乎未作探索。本文研究了将单模态模型适配至多模态任务所需的最小计算代价，并提出一种新的具有挑战性的实验设置及多种方法，以实现对单模态预训练模型的高效适配。实验表明，通过冻结超过99%的总参数、仅训练一个线性投影层、并仅前置一个可训练Token，我们的方法（简称eP-ALM）在遵循所提设置的前提下，于图像、视频和音频模态的VQA与描述生成任务中显著优于其他基线方法。代码开源地址：https://github.com/mshukor/eP-ALM。