Recent advancements in Large Language Models (LLMs) such as GPT4 have displayed exceptional multi-modal capabilities in following open-ended instructions given images. However, the performance of these models heavily relies on design choices such as network structures, training data, and training strategies, and these choices have not been extensively discussed in the literature, making it difficult to quantify progress in this field. To address this issue, this paper presents a systematic and comprehensive study, quantitatively and qualitatively, on training such models. We implement over 20 variants with controlled settings. Concretely, for network structures, we compare different LLM backbones and model designs. For training data, we investigate the impact of data and sampling strategies. For instructions, we explore the influence of diversified prompts on the instruction-following ability of the trained models. For benchmarks, we contribute the first, to our best knowledge, comprehensive evaluation set including both image and video tasks through crowd-sourcing. Based on our findings, we present Lynx, which performs the most accurate multi-modal understanding while keeping the best multi-modal generation ability compared to existing open-sourced GPT4-style models.

翻译：近年来，以GPT4为代表的大型语言模型在遵循图片引导的开放式指令方面展现出卓越的多模态能力。然而，这些模型的性能高度依赖于网络结构、训练数据和训练策略等设计选择，而现有文献对此缺乏系统性讨论，导致该领域的进展难以量化评估。针对这一问题，本文从定性与定量角度对这类模型的训练进行了系统全面的研究。我们实现了20余个受控设置的变体模型：在网络结构方面，比较了不同LLM骨干网络与模型设计；在训练数据方面，研究了数据配比与采样策略的影响；在指令设计方面，探索了多样化提示对训练模型指令遵循能力的作用；在基准测试方面，构建了业界首个（据我们所知）涵盖图像与视频任务的综合评估数据集（通过众包方式收集）。基于研究发现，我们提出了Lynx模型——与现有开源GPT4风格模型相比，该模型在保持最优多模态生成能力的同时，实现了最精准的多模态理解性能。