Imitating how humans move their gaze in a visual scene is a vital research problem for both visual understanding and psychology, kindling crucial applications such as building alive virtual characters. Previous studies aim to predict gaze trajectories when humans are free-viewing an image, searching for required targets, or looking for clues to answer questions in an image. While these tasks focus on visual-centric scenarios, humans move their gaze also along with audio signal inputs in more common scenarios. To fill this gap, we introduce a new task that predicts human gaze trajectories in a visual scene with synchronized audio inputs and provide a new dataset containing 20k gaze points from 8 subjects. To effectively integrate audio information and simulate the dynamic process of human gaze motion, we propose a novel learning framework called EyEar (Eye moving while Ear listening) based on physics-informed dynamics, which considers three key factors to predict gazes: eye inherent motion tendency, vision salient attraction, and audio semantic attraction. We also propose a probability density score to overcome the high individual variability of gaze trajectories, thereby improving the stabilization of optimization and the reliability of the evaluation. Experimental results show that EyEar outperforms all the baselines in the context of all evaluation metrics, thanks to the proposed components in the learning model.

翻译：模仿人类在视觉场景中如何移动视线，对于视觉理解和心理学都是一个至关重要的研究问题，并催生了构建生动虚拟角色等关键应用。先前的研究旨在预测人类在自由观看图像、搜索所需目标或在图像中寻找线索以回答问题时的注视轨迹。虽然这些任务聚焦于以视觉为中心的场景，但在更常见的场景中，人类的注视也会随着音频信号输入而移动。为填补这一空白，我们引入了一项新任务：预测在具有同步音频输入的视觉场景中的人类注视轨迹，并提供了一个包含来自8名受试者的2万个注视点的新数据集。为了有效整合音频信息并模拟人类注视运动的动态过程，我们提出了一种基于物理信息动力学的新型学习框架，称为EyEar（耳听目动）。该框架考虑了预测注视的三个关键因素：眼睛固有的运动趋势、视觉显著吸引和音频语义吸引。我们还提出了一种概率密度评分，以克服注视轨迹的高度个体差异性，从而提高优化的稳定性和评估的可靠性。实验结果表明，得益于学习模型中提出的各个组件，EyEar在所有评估指标的背景下均优于所有基线方法。