Gaze following and social gaze prediction are fundamental tasks providing insights into human communication behaviors, intent, and social interactions. Most previous approaches addressed these tasks separately, either by designing highly specialized social gaze models that do not generalize to other social gaze tasks or by considering social gaze inference as an ad-hoc post-processing of the gaze following task. Furthermore, the vast majority of gaze following approaches have proposed static models that can handle only one person at a time, therefore failing to take advantage of social interactions and temporal dynamics. In this paper, we address these limitations and introduce a novel framework to jointly predict the gaze target and social gaze label for all people in the scene. The framework comprises of: (i) a temporal, transformer-based architecture that, in addition to image tokens, handles person-specific tokens capturing the gaze information related to each individual; (ii) a new dataset, VSGaze, that unifies annotation types across multiple gaze following and social gaze datasets. We show that our model trained on VSGaze can address all tasks jointly, and achieves state-of-the-art results for multi-person gaze following and social gaze prediction.

翻译：视线跟随与社会性注视预测是理解人类交流行为、意图及社交互动的基础性任务。以往多数方法将这两个任务分开处理，要么设计高度专门化的社会性注视模型但无法泛化至其他社会性注视任务，要么将社会性注视推断作为视线跟随任务的临时后处理环节。此外，绝大多数视线跟随方法采用静态模型且每次仅能处理单个人物，因而无法利用社交互动与时序动态信息。针对上述局限性，本文提出一种新型框架，可联合预测场景中所有人的视线目标与社会性注视标签。该框架包含：（i）基于时序Transformer架构的处理单元，除图像令牌外，还可处理捕获个体注视信息的人物专属令牌；（ii）新型数据集VSGaze，统一了多个视线跟随与社会性注视数据集的标注类型。实验表明，在VSGaze上训练的模型能够联合处理所有任务，并在多人物视线跟随与社会性注视预测中达到了最先进水平。