Humans utilize their gaze to concentrate on essential information while perceiving and interpreting intentions in videos. Incorporating human gaze into computational algorithms can significantly enhance model performance in video understanding tasks. In this work, we address a challenging and innovative task in video understanding: predicting the actions of an agent in a video based on a partial video. We introduce the Gaze-guided Action Anticipation algorithm, which establishes a visual-semantic graph from the video input. Our method utilizes a Graph Neural Network to recognize the agent's intention and predict the action sequence to fulfill this intention. To assess the efficiency of our approach, we collect a dataset containing household activities generated in the VirtualHome environment, accompanied by human gaze data of viewing videos. Our method outperforms state-of-the-art techniques, achieving a 7\% improvement in accuracy for 18-class intention recognition. This highlights the efficiency of our method in learning important features from human gaze data.
翻译:人类在观看视频时通过目光聚焦于关键信息,同时感知和解读其中的意图。将人类目光信息融入计算算法可显著提升视频理解任务的模型性能。本文针对视频理解领域中的一项创新性挑战:基于部分视频片段预测智能体的动作序列。我们提出了目光引导动作预测算法,该算法从视频输入构建视觉语义图,利用图神经网络识别智能体意图并预测实现该意图的动作序列。为评估方法有效性,我们收集了包含虚拟家庭环境中生成的日常活动数据集,并附有人类观看视频时的目光数据。实验结果表明,本方法在18类意图识别任务中准确率较现有最优技术提升7%,充分验证了从人类目光数据中学习关键特征的有效性。