Evaluating visual activity recognition systems is challenging due to inherent ambiguities in verb semantics and image interpretation. When describing actions in images, synonymous verbs can refer to the same event (e.g., brushing vs. grooming), while different perspectives can lead to equally valid but distinct verb choices (e.g., piloting vs. operating). Standard exact-match evaluation, which relies on a single gold answer, fails to capture these ambiguities, resulting in an incomplete assessment of model performance. To address this, we propose a vision-language clustering framework that constructs verb sense clusters, providing a more robust evaluation. Our analysis of the imSitu dataset shows that each image maps to around four sense clusters, with each cluster representing a distinct perspective of the image. We evaluate multiple activity recognition models and compare our cluster-based evaluation with standard evaluation methods. Additionally, our human alignment analysis suggests that the cluster-based evaluation better aligns with human judgments, offering a more nuanced assessment of model performance.

翻译：评估视觉活动识别系统具有挑战性，这源于动词语义与图像解读固有的歧义性。在描述图像中的动作时，同义动词可能指向同一事件（例如，brushing 与 grooming），而不同视角可能导致同样有效但不同的动词选择（例如，piloting 与 operating）。依赖单一标准答案的标准精确匹配评估方法无法捕捉这些歧义，导致对模型性能的评估不完整。为解决这一问题，我们提出了一种视觉-语言聚类框架，该框架构建动词词义聚类，从而提供更鲁棒的评估。我们对 imSitu 数据集的分析表明，每幅图像映射到约四个词义聚类，每个聚类代表图像的一个独特视角。我们评估了多种活动识别模型，并将基于聚类的评估方法与标准评估方法进行了比较。此外，我们的人类对齐分析表明，基于聚类的评估能更好地与人类判断保持一致，从而为模型性能提供了更细致的评估。