Existing Open Vocabulary Detection (OVD) models exhibit a number of challenges. They often struggle with semantic consistency across diverse inputs, and are often sensitive to slight variations in input phrasing, leading to inconsistent performance. The calibration of their predictive confidence, especially in complex multi-label scenarios, remains suboptimal, frequently resulting in overconfident predictions that do not accurately reflect their context understanding. To understand these limitations, multi-label detection benchmarks are needed. A particularly challenging domain for such benchmarking is social activities. Due to the lack of multi-label benchmarks for social interactions, in this work we present ELSA: Evaluating Localization of Social Activities. ELSA draws on theoretical frameworks in urban sociology and design and uses in-the-wild street-level imagery, where the size of groups and the types of activities vary significantly. ELSA includes more than 900 manually annotated images with more than 4,300 multi-labeled bounding boxes for individual and group activities. We introduce a novel confidence score computation method NLSE and a novel Dynamic Box Aggregation (DBA) algorithm to assess semantic consistency in overlapping predictions. We report our results on the widely-used SOTA models Grounding DINO, Detic, OWL, and MDETR. Our evaluation protocol considers semantic stability and localization accuracy and further exposes the limitations of existing approaches.

翻译：现有的开放词汇检测（OVD）模型存在若干挑战。它们通常在多样化输入中难以保持语义一致性，并对输入表述的细微变化较为敏感，导致性能表现不稳定。其预测置信度的校准——尤其在复杂的多标签场景中——仍不理想，经常产生过度自信的预测，未能准确反映模型对上下文的理解。为深入理解这些局限性，需要建立多标签检测基准。社交活动领域是此类基准测试中尤为困难的范畴。由于缺乏针对社交互动的多标签基准，本研究提出ELSA：社交活动定位评估基准。ELSA借鉴城市社会学与设计领域的理论框架，采用真实场景的街景图像数据，其中群体规模与活动类型存在显著差异。该数据集包含900余张人工标注图像，涵盖超过4,300个针对个体与群体活动的多标签边界框。我们提出了一种新颖的置信度计算方法NLSE以及创新的动态边界框聚合（DBA）算法，用于评估重叠预测中的语义一致性。我们在广泛使用的先进模型Grounding DINO、Detic、OWL和MDETR上报告了实验结果。我们的评估方案综合考虑语义稳定性与定位准确性，进一步揭示了现有方法的局限性。