Referring expression comprehension (REC) involves localizing a target instance based on a textual description. Recent advancements in REC have been driven by large multimodal models (LMMs) like CogVLM, which achieved 92.44% accuracy on RefCOCO. However, this study questions whether existing benchmarks such as RefCOCO, RefCOCO+, and RefCOCOg, capture LMMs' comprehensive capabilities. We begin with a manual examination of these benchmarks, revealing high labeling error rates: 14% in RefCOCO, 24% in RefCOCO+, and 5% in RefCOCOg, which undermines the authenticity of evaluations. We address this by excluding problematic instances and reevaluating several LMMs capable of handling the REC task, showing significant accuracy improvements, thus highlighting the impact of benchmark noise. In response, we introduce Ref-L4, a comprehensive REC benchmark, specifically designed to evaluate modern REC models. Ref-L4 is distinguished by four key features: 1) a substantial sample size with 45,341 annotations; 2) a diverse range of object categories with 365 distinct types and varying instance scales from 30 to 3,767; 3) lengthy referring expressions averaging 24.2 words; and 4) an extensive vocabulary comprising 22,813 unique words. We evaluate a total of 24 large models on Ref-L4 and provide valuable insights. The cleaned versions of RefCOCO, RefCOCO+, and RefCOCOg, as well as our Ref-L4 benchmark and evaluation code, are available at https://github.com/JierunChen/Ref-L4.

翻译：指代表达理解（REC）涉及根据文本描述定位目标实例。近期REC的进展由大型多模态模型（LMMs）如CogVLM推动，其在RefCOCO上达到了92.44%的准确率。然而，本研究质疑现有基准测试（如RefCOCO、RefCOCO+和RefCOCOg）是否充分捕捉了LMMs的综合能力。我们首先对这些基准进行了人工检查，揭示了较高的标注错误率：RefCOCO为14%，RefCOCO+为24%，RefCOCOg为5%，这损害了评估的真实性。我们通过排除问题实例并重新评估多个能够处理REC任务的LMMs来解决这一问题，结果显示准确率显著提升，从而突显了基准噪声的影响。为此，我们引入了Ref-L4，一个专门设计用于评估现代REC模型的综合性REC基准。Ref-L4具有四个关键特征：1) 大规模样本量，包含45,341个标注；2) 多样化的对象类别，涵盖365种不同类型及实例尺度从30到3,767不等；3) 长篇幅的指代表达，平均长度为24.2个词；4) 广泛的词汇表，包含22,813个独特词汇。我们在Ref-L4上评估了总计24个大型模型，并提供了有价值的见解。RefCOCO、RefCOCO+和RefCOCOg的清理版本，以及我们的Ref-L4基准和评估代码，可在https://github.com/JierunChen/Ref-L4获取。