Large Language Models (LLMs) have so far impressed the world, with unprecedented capabilities that emerge in models at large scales. On the vision side, transformer models (i.e., ViT) are following the same trend, achieving the best performance on challenging benchmarks. With the abundance of such unimodal models, a natural question arises; do we need also to follow this trend to tackle multimodal tasks? In this work, we propose to rather direct effort to efficient adaptations of existing models, and propose to augment Language Models with perception. Existing approaches for adapting pretrained models for vision-language tasks still rely on several key components that hinder their efficiency. In particular, they still train a large number of parameters, rely on large multimodal pretraining, use encoders (e.g., CLIP) trained on huge image-text datasets, and add significant inference overhead. In addition, most of these approaches have focused on Zero-Shot and In Context Learning, with little to no effort on direct finetuning. We investigate the minimal computational effort needed to adapt unimodal models for multimodal tasks and propose a new challenging setup, alongside different approaches, that efficiently adapts unimodal pretrained models. We show that by freezing more than 99% of total parameters, training only one linear projection layer, and prepending only one trainable token, our approach (dubbed eP-ALM) significantly outperforms other baselines on VQA and Captioning across Image, Video, and Audio modalities, following the proposed setup. The code is available here: https://github.com/mshukor/eP-ALM.

翻译：大型语言模型（LLMs）以其在大规模模型中涌现出的前所未有的能力令世界瞩目。在视觉领域，Transformer模型（如ViT）也遵循同样的趋势，在具有挑战性的基准测试中取得了最佳性能。在如此丰富的单模态模型背景下，一个自然的问题随之产生：我们是否也需要遵循这一趋势来处理多模态任务？本文提出将工作重心转向对现有模型的高效适配，并主张通过感知能力增强语言模型。现有的预训练模型适配方法在视觉-语言任务中仍依赖于若干关键组件，这些组件限制了其效率：它们仍需训练大量参数、依赖大规模多模态预训练、使用在海量图像-文本数据集上训练的编码器（如CLIP），并增加了显著的推理开销。此外，这些方法大多聚焦于零样本学习和上下文学习，而在直接微调方面的投入甚微。我们探究了将单模态模型适配至多模态任务所需的最小计算量，并提出一种新的挑战性设置及多种方法，以实现对单模态预训练模型的高效适配。实验表明，通过冻结超过99%的总参数、仅训练一个线性投影层并添加一个可训练令牌，我们的方法（称为eP-ALM）在遵循所提出设置的前提下，在图像、视频和音频模态下的VQA（视觉问答）与描述生成任务中显著优于其他基线方法。代码已开源：https://github.com/mshukor/eP-ALM。