In this paper, we highlight a problem of evaluation metrics adopted in the open-vocabulary segmentation. That is, the evaluation process still heavily relies on closed-set metrics on zero-shot or cross-dataset pipelines without considering the similarity between predicted and ground truth categories. To tackle this issue, we first survey eleven similarity measurements between two categorical words using WordNet linguistics statistics, text embedding, and language models by comprehensive quantitative analysis and user study. Built upon those explored measurements, we designed novel evaluation metrics, namely Open mIoU, Open AP, and Open PQ, tailored for three open-vocabulary segmentation tasks. We benchmarked the proposed evaluation metrics on 12 open-vocabulary methods of three segmentation tasks. Even though the relative subjectivity of similarity distance, we demonstrate that our metrics can still well evaluate the open ability of the existing open-vocabulary segmentation methods. We hope that our work can bring with the community new thinking about how to evaluate the open ability of models. The evaluation code is released in github.

翻译：本文指出了开放词汇分割中采用的评估指标存在的问题。当前评估流程在零样本或跨数据集流水线上仍严重依赖封闭集指标，未考虑预测类别与真实类别之间的语义相似性。为解决这一问题，我们首先通过全面的定量分析与用户研究，利用WordNet语言学统计、文本嵌入和语言模型对两个类别词之间的十一种相似度度量方法进行了调研。基于这些探索出的度量方法，我们针对三种开放词汇分割任务设计了新型评估指标——Open mIoU、Open AP和Open PQ。我们在三种分割任务的12种开放词汇方法上对提出的评估指标进行了基准测试。尽管相似性距离具有相对主观性，但实验表明我们的指标仍能有效评估现有开放词汇分割方法的开放能力。希望我们的工作能为学界带来关于如何评估模型开放能力的新思考。评估代码已在GitHub上发布。