We introduce VisIT-Bench (Visual InsTruction Benchmark), a benchmark for evaluation of instruction-following vision-language models for real-world use. Our starting point is curating 70 'instruction families' that we envision instruction tuned vision-language models should be able to address. Extending beyond evaluations like VQAv2 and COCO, tasks range from basic recognition to game playing and creative generation. Following curation, our dataset comprises 592 test queries, each with a human-authored instruction-conditioned caption. These descriptions surface instruction-specific factors, e.g., for an instruction asking about the accessibility of a storefront for wheelchair users, the instruction-conditioned caption describes ramps/potential obstacles. These descriptions enable 1) collecting human-verified reference outputs for each instance; and 2) automatic evaluation of candidate multimodal generations using a text-only LLM, aligning with human judgment. We quantify quality gaps between models and references using both human and automatic evaluations; e.g., the top-performing instruction-following model wins against the GPT-4 reference in just 27% of the comparison. VisIT-Bench is dynamic to participate, practitioners simply submit their model's response on the project website; Data, code and leaderboard is available at visit-bench.github.io.

翻译：我们引入VisIT-Bench（视觉指令基准），这是一个用于评估面向真实世界应用的指令跟随视觉语言模型的基准。我们的出发点是以精心设计的70个“指令族”为核心，这些指令族被设想为指令微调后的视觉语言模型应能处理的任务。该基准超越VQAv2和COCO等评估范畴，涵盖从基础识别到游戏互动及创意生成等多样化任务。经整理，数据集包含592个测试查询，每个查询附有人工编写的指令条件化描述。这些描述揭示了指令特定的因素，例如，对于询问店铺入口对轮椅使用者无障碍程度的指令，条件化描述会注明斜坡/潜在障碍物。此类描述能够：1）为每个实例收集人工验证的参考输出；2）利用纯文本大语言模型自动评估候选多模态生成结果，并与人类判断对齐。我们通过人工和自动评估量化模型与参考答案之间的质量差距。例如，性能最佳的指令跟随模型在与GPT-4参考输出的对比中，胜率仅为27%。VisIT-Bench支持动态参与，从业者只需在项目网站提交模型响应即可；数据、代码及排行榜可通过visit-bench.github.io获取。