Instruction tuning large language models (LLMs) using machine-generated instruction-following data has improved zero-shot capabilities on new tasks, but the idea is less explored in the multimodal field. In this paper, we present the first attempt to use language-only GPT-4 to generate multimodal language-image instruction-following data. By instruction tuning on such generated data, we introduce LLaVA: Large Language and Vision Assistant, an end-to-end trained large multimodal model that connects a vision encoder and LLM for general-purpose visual and language understanding.Our early experiments show that LLaVA demonstrates impressive multimodel chat abilities, sometimes exhibiting the behaviors of multimodal GPT-4 on unseen images/instructions, and yields a 85.1% relative score compared with GPT-4 on a synthetic multimodal instruction-following dataset. When fine-tuned on Science QA, the synergy of LLaVA and GPT-4 achieves a new state-of-the-art accuracy of 92.53%. We make GPT-4 generated visual instruction tuning data, our model and code base publicly available.

翻译：通过使用机器生成的指令遵循数据对大型语言模型（LLM）进行指令微调，已提升了模型在新任务上的零样本能力，但这一思想在多模态领域尚未得到充分探索。本文首次尝试仅利用基于语言的GPT-4生成多模态语言-图像指令遵循数据。通过对此类生成数据进行指令微调，我们提出了LLaVA：大型语言与视觉助手——一种端到端训练的大型多模态模型，该模型连接视觉编码器与LLM，以实现通用的视觉与语言理解。初步实验表明，LLaVA展现出令人印象深刻的多模态对话能力，有时能在未见图像/指令上呈现出多模态GPT-4的行为特征，并在合成多模态指令遵循数据集上达到GPT-4相对分数的85.1%。当在Science QA上进行微调时，LLaVA与GPT-4的协同作用实现了92.53%的新最先进准确率。我们将GPT-4生成的视觉指令微调数据、模型及代码库公开发布。