Large Language Models (LLMs) have so far impressed the world, with unprecedented capabilities that emerge in models at large scales. On the vision side, transformer models (i.e., ViT) are following the same trend, achieving the best performance on challenging benchmarks. With the abundance of such unimodal models, a natural question arises; do we need also to follow this trend to tackle multimodal tasks? In this work, we propose to rather direct effort to efficient adaptations of existing models, and propose to augment Language Models with perception. Existing approaches for adapting pretrained models for vision-language tasks still rely on several key components that hinder their efficiency. In particular, they still train a large number of parameters, rely on large multimodal pretraining, use encoders (e.g., CLIP) trained on huge image-text datasets, and add significant inference overhead. In addition, most of these approaches have focused on Zero-Shot and In Context Learning, with little to no effort on direct finetuning. We investigate the minimal computational effort needed to adapt unimodal models for multimodal tasks and propose a new challenging setup, alongside different approaches, that efficiently adapts unimodal pretrained models. We show that by freezing more than 99\% of total parameters, training only one linear projection layer, and prepending only one trainable token, our approach (dubbed eP-ALM) significantly outperforms other baselines on VQA and Captioning across Image, Video, and Audio modalities, following the proposed setup. The code will be available here: https://github.com/mshukor/eP-ALM.

翻译：大型语言模型（LLMs）在规模化模型中展现出前所未有的能力，已令世界瞩目。在视觉方面，Transformer模型（如ViT）正遵循相同趋势，在具有挑战性的基准测试中取得了最佳性能。面对此类丰富单模态模型的存在，一个自然的问题随之而来：我们是否也必须遵循这一趋势来应对多模态任务？在本工作中，我们提出将努力方向转向对现有模型的高效适配，并提议增强语言模型的感知能力。现有用于适配预训练模型以完成视觉-语言任务的方法，仍依赖于若干阻碍其效率的关键组件。具体而言，这些方法仍需训练大量参数、依赖大规模多模态预训练、使用基于海量图像-文本数据集训练的编码器（如CLIP），并增加显著推理开销。此外，大多数此类方法聚焦于零样本学习和上下文学习，而对直接微调的投入极少。我们探究了为多模态任务适配单模态模型所需的最小计算开销，并提出了一种新的具有挑战性的设置及多种方法，以实现对单模态预训练模型的高效适配。实验表明，通过冻结超过99%的总参数、仅训练一个线性投影层并仅预置一个可训练令牌，我们的方法（称为eP-ALM）在遵循所提设置的条件下，于图像、视频和音频模态的VQA和标题生成任务中显著优于其他基线方法。代码将在此公开：https://github.com/mshukor/eP-ALM。