We introduce VisIT-Bench (Visual InsTruction Benchmark), a benchmark for evaluation of instruction-following vision-language models for real-world use. Our starting point is curating 70 'instruction families' that we envision instruction tuned vision-language models should be able to address. Extending beyond evaluations like VQAv2 and COCO, tasks range from basic recognition to game playing and creative generation. Following curation, our dataset comprises 592 test queries, each with a human-authored instruction-conditioned caption. These descriptions surface instruction-specific factors, e.g., for an instruction asking about the accessibility of a storefront for wheelchair users, the instruction-conditioned caption describes ramps/potential obstacles. These descriptions enable 1) collecting human-verified reference outputs for each instance; and 2) automatic evaluation of candidate multimodal generations using a text-only LLM, aligning with human judgment. We quantify quality gaps between models and references using both human and automatic evaluations; e.g., the top-performing instruction-following model wins against the GPT-4 reference in just 27% of the comparison. VisIT-Bench is dynamic to participate, practitioners simply submit their model's response on the project website; Data, code and leaderboard is available at visit-bench.github.io.

翻译：我们提出VisIT-Bench（视觉指令基准），这是一个面向真实世界应用的指令跟随视觉语言模型评估基准。我们从策划70个"指令家族"出发——这些是我们期望指令调优后的视觉语言模型应具备的处理能力。相较于VQAv2和COCO等评估基准，本任务涵盖从基础识别到游戏操作和创意生成等多元化范畴。经过策划后，数据集包含592个测试查询，每个查询附带人工撰写的指令条件化描述。这些描述揭示指令特定因素，例如针对询问商店入口轮椅无障碍性的指令，其条件化描述会注明坡道/潜在障碍物。这些描述可实现：1）为每个实例收集人工验证的参考输出；2）利用纯文本大语言模型对候选多模态生成结果进行与人类判断对齐的自动评估。我们通过人工与自动评估量化模型与参考输出之间的质量差距——例如，表现最优的指令跟随模型仅在27%的比较中胜出GPT-4参考输出。VisIT-Bench支持动态参与，从业者只需在项目网站提交模型响应；数据、代码和排行榜可在visit-bench.github.io获取。