Instruction-tuned large language models (LLMs) have demonstrated promising zero-shot generalization capabilities across various downstream tasks. Recent research has introduced multimodal capabilities to LLMs by integrating independently pretrained vision encoders through model grafting. These multimodal variants undergo instruction tuning, similar to LLMs, enabling effective zero-shot generalization for multimodal tasks. This study conducts a comparative analysis of different multimodal instruction tuning approaches and evaluates their performance across a range of tasks, including complex reasoning, conversation, image captioning, multiple-choice questions (MCQs), and binary classification. Through rigorous benchmarking and ablation experiments, we reveal key insights for guiding architectural choices when incorporating multimodal capabilities into LLMs. However, current approaches have limitations; they do not sufficiently address the need for a diverse multimodal instruction dataset, which is crucial for enhancing task generalization. Additionally, they overlook issues related to truthfulness and factuality when generating responses. These findings illuminate current methodological constraints in adapting language models for image comprehension and provide valuable guidance for researchers and practitioners seeking to harness multimodal versions of LLMs.

翻译：指令微调的大语言模型（LLMs）在各类下游任务中展现了出色的零样本泛化能力。近期研究通过模型嫁接技术，将独立预训练的视觉编码器与LLMs整合，为其赋予了多模态能力。这些多模态变体与LLMs类似，经过指令微调后能够有效实现多模态任务的零样本泛化。本研究对不同多模态指令微调方法进行了比较分析，并评估了它们在包括复杂推理、对话、图像描述、多项选择题和二元分类等一系列任务上的性能。通过严格的基准测试和消融实验，我们揭示了在LLMs中融入多模态能力时指导架构选择的关键见解。然而，当前方法存在局限性：它们未能充分满足对多样化多模态指令数据集的需求，而这对于增强任务泛化能力至关重要。此外，这些方法在生成响应时忽视了真实性和事实性问题。这些发现阐明了当前在使语言模型适应图像理解方面的方法论限制，并为致力于利用LLMs多模态版本的研究人员和从业者提供了宝贵指导。