The growing interest in vision-language models (VLMs) has been driven by improvements in large language models and vision transformers. Despite the abundance of literature on this subject, we observe that critical decisions regarding the design of VLMs are often not justified. We argue that these unsupported decisions impede progress in the field by making it difficult to identify which choices improve model performance. To address this issue, we conduct extensive experiments around pre-trained models, architecture choice, data, and training methods. Our consolidation of findings includes the development of Idefics2, an efficient foundational VLM of 8 billion parameters. Idefics2 achieves state-of-the-art performance within its size category across various multimodal benchmarks, and is often on par with models four times its size. We release the model (base, instructed, and chat) along with the datasets created for its training.

翻译：视觉语言模型（VLM）的兴起得益于大语言模型和视觉Transformer的进步。尽管相关文献数量众多，但我们观察到，关于VLM设计的关键决策往往缺乏充分论证。我们认为，这些未经证实的决策阻碍了该领域的发展，使研究者难以辨别哪些选择能真正提升模型性能。为解决这一问题，我们围绕预训练模型、架构选择、数据及训练方法开展了大规模实验。通过整合研究成果，我们开发了Idefics2——一个高效的80亿参数基础视觉语言模型。Idefics2在其尺寸类别中，于多项多模态基准测试中达到最优性能，且常与四倍于其规模的模型表现相当。我们已公开发布该模型（包括基础版、指令版和对话版）及训练所用的数据集。