Assessing long-form responses generated by Vision-Language Models (VLMs) is challenging. It not only requires checking whether the VLM follows the given instruction but also verifying whether the text output is properly grounded on the given image. Inspired by the recent approach of evaluating LMs with LMs, in this work, we propose to evaluate VLMs with VLMs. For this purpose, we present a new feedback dataset called the Perception Collection, encompassing 15K customized score rubrics that users might care about during assessment. Using the Perception Collection, we train Prometheus-Vision, the first open-source VLM evaluator model that can understand the user-defined score criteria during evaluation. Prometheus-Vision shows the highest Pearson correlation with human evaluators and GPT-4V among open-source models, showing its effectiveness for transparent and accessible evaluation of VLMs. We open-source our code, dataset, and model at https://github.com/kaistAI/prometheus-vision

翻译：评估视觉语言模型（VLM）生成的长格式响应具有挑战性。这不仅需要检查VLM是否遵循给定指令，还需验证文本输出是否合理基于输入图像。受近期以语言模型评估语言模型方法的启发，本工作提出以VLM评估VLM方案。为此，我们构建了名为感知集合（Perception Collection）的新型反馈数据集，包含15,000条用户可能在评估中关注的定制化评分标准。基于感知集合，我们训练了Prometheus-Vision——首个能理解用户自定义评分标准的开源VLM评估模型。Prometheus-Vision在开源模型中与人类评估者和GPT-4V的皮尔逊相关系数最高，展示了其对VLM进行透明可评估的有效性。我们在https://github.com/kaistAI/prometheus-vision 开源了代码、数据集及模型。