We introduce VisIT-Bench (Visual InsTruction Benchmark), a benchmark for evaluation of instruction-following vision-language models for real-world use. Our starting point is curating 70 'instruction families' that we envision instruction tuned vision-language models should be able to address. Extending beyond evaluations like VQAv2 and COCO, tasks range from basic recognition to game playing and creative generation. Following curation, our dataset comprises 592 test queries, each with a human-authored instruction-conditioned caption. These descriptions surface instruction-specific factors, e.g., for an instruction asking about the accessibility of a storefront for wheelchair users, the instruction-conditioned caption describes ramps/potential obstacles. These descriptions enable 1) collecting human-verified reference outputs for each instance; and 2) automatic evaluation of candidate multimodal generations using a text-only LLM, aligning with human judgment. We quantify quality gaps between models and references using both human and automatic evaluations; e.g., the top-performing instruction-following model wins against the GPT-4 reference in just 27% of the comparison. VisIT-Bench is dynamic to participate, practitioners simply submit their model's response on the project website; Data, code and leaderboard is available at visit-bench.github.io.

翻译：我们提出了VisIT-Bench（视觉指令基准），这是一个用于评估面向真实世界应用的指令跟随视觉语言模型的基准。我们的起点是策划70个“指令族”，我们认为经过指令调优的视觉语言模型应能处理这些任务。评估范围超越了VQAv2和COCO等传统基准，任务涵盖从基础识别到游戏交互及创意生成。在策划完成后，我们的数据集包含592个测试查询，每个查询附有人工撰写的指令条件化描述。这些描述揭示了指令特定因素，例如，对于询问店面轮椅无障碍性的指令，条件化描述会标注坡道/潜在障碍物。这些描述使我们能够：1）为每个实例收集人工验证的参考答案；2）使用纯文本大语言模型对候选多模态生成结果进行自动评估，并与人类判断对齐。我们通过人类和自动评估量化了模型与参考答案之间的质量差距；例如，在对比中，表现最佳的指令跟随模型仅在27%的情况下胜出GPT-4参考答案。VisIT-Bench支持动态参与，研究者只需在项目网站上提交其模型响应即可；数据、代码及排行榜可访问visit-bench.github.io获取。