Large Language Models (LLMs) have so far impressed the world, with unprecedented capabilities that emerge in models at large scales. On the vision side, transformer models (i.e., ViT) are following the same trend, achieving the best performance on challenging benchmarks. With the abundance of such unimodal models, a natural question arises; do we need also to follow this trend to tackle multimodal tasks? In this work, we propose to rather direct effort to efficient adaptations of existing models, and propose to augment Language Models with perception. Existing approaches for adapting pretrained models for vision-language tasks still rely on several key components that hinder their efficiency. In particular, they still train a large number of parameters, rely on large multimodal pretraining, use encoders (e.g., CLIP) trained on huge image-text datasets, and add significant inference overhead. In addition, most of these approaches have focused on Zero-Shot and In Context Learning, with little to no effort on direct finetuning. We investigate the minimal computational effort needed to adapt unimodal models for multimodal tasks and propose a new challenging setup, alongside different approaches, that efficiently adapts unimodal pretrained models. We show that by freezing more than 99\% of total parameters, training only one linear projection layer, and prepending only one trainable token, our approach (dubbed eP-ALM) significantly outperforms other baselines on VQA and Captioning across Image, Video, and Audio modalities, following the proposed setup. The code will be available here: https://github.com/mshukor/eP-ALM.

翻译：大型语言模型（LLMs）以其在大规模模型中涌现出的前所未有的能力给世界留下了深刻印象。在视觉方面，Transformer模型（如ViT）也遵循着同样的趋势，在具有挑战性的基准测试中取得了最佳性能。鉴于此类单一模态模型的丰富性，一个自然的问题随之产生：我们是否也需要遵循这一趋势来处理多模态任务？在本文中，我们建议将精力转向对现有模型的高效适配，并提出增强语言模型感知能力的方法。现有将预训练模型适配到视觉-语言任务的方法仍依赖于若干阻碍其效率的关键组件。具体而言，它们仍需训练大量参数、依赖大规模多模态预训练、使用在大型图像-文本数据集上训练的编码器（如CLIP）以及引入显著的推理开销。此外，这些方法大多专注于零样本学习和上下文学习，而在直接微调方面的努力微乎其微。我们研究了将单一模态模型适配到多模态任务所需的最小计算代价，并提出了一种新的具有挑战性的设定及相关方法，以高效适配单模态预训练模型。结果表明，通过冻结超过99%的总参数、仅训练一个线性投影层并仅预置一个可训练标记，我们的方法（称为eP-ALM）在遵循所提设定的情况下，在涉及图像、视频和音频模态的VQA与描述生成任务上显著优于其他基线方法。代码将在此处提供：https://github.com/mshukor/eP-ALM。