Recent advancements in Large Language Models (LLMs) such as GPT4 have displayed exceptional multi-modal capabilities in following open-ended instructions given images. However, the performance of these models heavily relies on design choices such as network structures, training data, and training strategies, and these choices have not been extensively discussed in the literature, making it difficult to quantify progress in this field. To address this issue, this paper presents a systematic and comprehensive study, quantitatively and qualitatively, on training such models. We implement over 20 variants with controlled settings. Concretely, for network structures, we compare different LLM backbones and model designs. For training data, we investigate the impact of data and sampling strategies. For instructions, we explore the influence of diversified prompts on the instruction-following ability of the trained models. For benchmarks, we contribute the first, to our best knowledge, comprehensive evaluation set including both image and video tasks through crowd-sourcing. Based on our findings, we present Lynx, which performs the most accurate multi-modal understanding while keeping the best multi-modal generation ability compared to existing open-sourced GPT4-style models.

翻译：近期以GPT4为代表的大型语言模型在遵循图像开放指令方面展现了卓越的多模态能力。然而，这些模型的性能高度依赖于网络结构、训练数据及训练策略等设计选择，而现有文献对这些选择缺乏系统性讨论，导致该领域的进展难以量化评估。为解决这一问题，本文通过定量与定性相结合的方式，对这类模型的训练进行了系统全面的研究。我们实现了20余种受控设置的变体模型。具体而言：在网络结构层面，比较了不同LLM骨干网络与模型设计方案；在训练数据层面，探究了数据构成与采样策略的影响；在指令设计层面，探索了多样化提示对训练模型指令遵循能力的作用；在基准测试层面，首次（据我们所知）通过众包方式构建了涵盖图像与视频任务的综合评估集。基于研究结果，我们提出了Lynx模型——与现有开源GPT4风格模型相比，该模型在保持最优多模态生成能力的同时，实现了最精准的多模态理解性能。