Social group detection, or the identification of humans involved in reciprocal interpersonal interactions (e.g., family members, friends, and customers and merchants), is a crucial component of social intelligence needed for agents transacting in the world. The few existing benchmarks for social group detection are limited by low scene diversity and reliance on third-person camera sources (e.g., surveillance footage). Consequently, these benchmarks generally lack real-world evaluation on how groups form and evolve in diverse cultural contexts and unconstrained settings. To address this gap, we introduce EgoGroups, a first-person view dataset that captures social dynamics in cities around the world. EgoGroups spans 65 countries covering low, medium, and high-crowd settings under four weather/time-of-day conditions. We include dense human annotations for person and social groups, along with rich geographic and scene metadata. Using this dataset, we performed an extensive evaluation of state-of-the-art VLM/LLMs and supervised models on their group detection capabilities. We found several interesting findings, including VLMs and LLMs can outperform supervised baselines in a zero-shot setting, while crowd density and cultural regions clearly influence model performance.

翻译：社交群体检测，即识别参与互惠人际互动的人类个体（如家庭成员、朋友、顾客与商家），是智能体进行社会交互所需的核心社会智能。现有少数社交群体检测基准数据集存在场景多样性不足、依赖第三方视角（如监控录像）等局限。因此，这些基准数据集普遍缺乏对群体在不同文化背景与无约束环境下形成与演化的真实世界评估。为填补这一空白，我们提出了EgoGroups——一个捕捉全球城市社交动态的第一人称视角数据集。该数据集覆盖65个国家，包含低、中、高人群密度场景，涵盖四种天气/时段条件。我们为人物及社交群体标注了密集的人工标注信息，并附有丰富的地理与场景元数据。基于该数据集，我们对当前最先进的视觉语言模型/大语言模型及监督模型的群体检测能力进行了全面评估。研究发现若干有趣现象：包括视觉语言模型与大语言模型在零样本设置下可超越监督基线模型，同时人群密度与文化区域显著影响模型性能。