Visual instruction tuning is an essential approach to improving the zero-shot generalization capability of Multi-modal Large Language Models (MLLMs). A surge of visual instruction datasets with various focuses and characteristics have been proposed recently, enabling MLLMs to achieve surprising results on evaluation benchmarks. To develop more capable MLLMs, in this paper, we aim to investigate a more fundamental question: ``what makes for good visual instructions?''. By conducting a comprehensive empirical study, we find that instructions focused on complex visual reasoning tasks are particularly effective in improving the performance of MLLMs on evaluation benchmarks. Building upon this finding, we design a systematic approach to automatically creating high-quality complex visual reasoning instructions. Our approach employs a synthesis-complication-reformulation paradigm, leveraging multiple stages to gradually increase the complexity of the instructions while guaranteeing quality. Based on this approach, we create the synthetic visual reasoning instruction dataset consisting of 32K examples, namely ComVint, and fine-tune four MLLMs on it. Experimental results demonstrate that our dataset consistently enhances the performance of all the compared MLLMs, e.g., improving the performance of MiniGPT-4 and BLIP-2 on MME-Cognition by 32.6% and 28.8%, respectively. Our code and data are publicly available at the link: https://github.com/RUCAIBox/ComVint.

翻译：视觉指令微调是提升多模态大语言模型（MLLMs）零样本泛化能力的关键途径。近年来涌现出大量具有不同侧重点和特性的视觉指令数据集，使MLLMs在评估基准上取得了令人瞩目的成果。为开发更强大的MLLM，本文旨在探究一个更根本的问题：“何为优秀的视觉指令？”通过全面的实证研究，我们发现专注于复杂视觉推理任务的指令在提升MLLM评估基准性能方面尤为有效。基于此发现，我们设计了一套系统性方法，用于自动创建高质量的复杂视觉推理指令。该方法采用“合成-复杂化-重构”范式，通过多阶段流程逐步提升指令复杂度，同时确保质量。基于该范式，我们构建了包含32K样本的合成视觉推理指令数据集ComVint，并在此数据集上微调了四种MLLM。实验结果表明，该数据集持续提升了所有对比MLLM的性能，例如使MiniGPT-4和BLIP-2在MME-Cognition上的性能分别提升32.6%和28.8%。我们的代码与数据已公开于：https://github.com/RUCAIBox/ComVint。