There is an emerging line of research on multimodal instruction tuning, and a line of benchmarks have been proposed for evaluating these models recently. Instead of evaluating the models directly, in this paper we try to evaluate the Vision-Language Instruction-Tuning (VLIT) datasets themselves and further seek the way of building a dataset for developing an all-powerful VLIT model, which we believe could also be of utility for establishing a grounded protocol for benchmarking VLIT models. For effective analysis of VLIT datasets that remains an open question, we propose a tune-cross-evaluation paradigm: tuning on one dataset and evaluating on the others in turn. For each single tune-evaluation experiment set, we define the Meta Quality (MQ) as the mean score measured by a series of caption metrics including BLEU, METEOR, and ROUGE-L to quantify the quality of a certain dataset or a sample. On this basis, to evaluate the comprehensiveness of a dataset, we develop the Dataset Quality (DQ) covering all tune-evaluation sets. To lay the foundation for building a comprehensive dataset and developing an all-powerful model for practical applications, we further define the Sample Quality (SQ) to quantify the all-sided quality of each sample. Extensive experiments validate the rationality of the proposed evaluation paradigm. Based on the holistic evaluation, we build a new dataset, REVO-LION (REfining VisiOn-Language InstructiOn tuNing), by collecting samples with higher SQ from each dataset. With only half of the full data, the model trained on REVO-LION can achieve performance comparable to simply adding all VLIT datasets up. In addition to developing an all-powerful model, REVO-LION also includes an evaluation set, which is expected to serve as a convenient evaluation benchmark for future research.

翻译：多模态指令微调正成为新兴研究方向，近期涌现出大量用于评估此类模型的基准测试。与直接评估模型不同，本文致力于评估视觉-语言指令微调（VLIT）数据集本身，并进一步探索构建通用型VLIT模型的数据集建设方法——我们相信这亦有助于建立标准化的VLIT模型评估协议。针对目前尚属空白的VLIT数据集有效分析问题，我们提出交叉微调-评估范式：依次在单个数据集上进行微调，并在其余数据集上进行评估。针对每组微调-评估实验，我们定义元质量（MQ）为通过BLEU、METEOR和ROUGE-L等系列描述指标测量的平均得分，用以量化特定数据集或样本的质量。在此基础上，为评估数据集的全面性，我们构建覆盖所有微调-评估组合的数据集质量（DQ）。为奠定构建全面数据集和开发实用通用模型的基础，我们进一步定义样本质量（SQ）以量化每个样本的全方位质量。大量实验验证了所提评估范式的合理性。基于整体评估结果，我们从各数据集中筛选高SQ样本构建新数据集REVO-LION（视觉-语言指令微调精炼数据集）。仅需全量数据的一半规模，基于REVO-LION训练的模型即可达到简单合并所有VLIT数据集时的性能水平。除开发通用型模型外，REVO-LION还包含评估子集，有望成为未来研究的便捷评估基准。