We introduce VisIT-Bench (Visual InsTruction Benchmark), a benchmark for evaluation of instruction-following vision-language models for real-world use. Our starting point is curating 70 'instruction families' that we envision instruction tuned vision-language models should be able to address. Extending beyond evaluations like VQAv2 and COCO, tasks range from basic recognition to game playing and creative generation. Following curation, our dataset comprises 592 test queries, each with a human-authored instruction-conditioned caption. These descriptions surface instruction-specific factors, e.g., for an instruction asking about the accessibility of a storefront for wheelchair users, the instruction-conditioned caption describes ramps/potential obstacles. These descriptions enable 1) collecting human-verified reference outputs for each instance; and 2) automatic evaluation of candidate multimodal generations using a text-only LLM, aligning with human judgment. We quantify quality gaps between models and references using both human and automatic evaluations; e.g., the top-performing instruction-following model wins against the GPT-4 reference in just 27% of the comparison. VisIT-Bench is dynamic to participate, practitioners simply submit their model's response on the project website; Data, code and leaderboard is available at visit-bench.github.io.

翻译：我们提出VisIT-Bench（视觉指令基准测试），这是一个针对真实世界应用的指令跟随视觉语言模型评估基准。首先，我们构建了70个“指令族”，这些指令族是期望经指令微调的视觉语言模型应能处理的任务范畴。该基准超越了VQAv2和COCO等传统评估，任务范围从基础识别延伸至游戏操作与创意生成。经过数据筛选，最终数据集包含592个测试查询，每个查询配备人工撰写的指令条件化描述。这些描述揭示了指令特定因素，例如面对“评估店铺入口对轮椅用户的通行便利性”的指令时，描述会标注坡道/潜在障碍物。这些描述可实现：1）为每个实例收集人工验证的参考输出；2）利用纯文本大语言模型对候选多模态生成结果进行自动化评估，且评估结果与人工判断高度一致。我们通过人工与自动评估量化了模型与参考输出间的质量差距——例如，最优指令跟随模型在与GPT-4参考输出的对比中，仅27%的案例胜出。VisIT-Bench支持动态参与，研究者仅需在项目网站提交模型响应；相关数据、代码与排行榜请访问visit-bench.github.io。