Situated conversations, which refer to visual information as visual question answering (VQA), often contain ambiguities caused by reliance on directive information. This problem is exacerbated because some languages, such as Japanese, often omit subjective or objective terms. Such ambiguities in questions are often clarified by the contexts in conversational situations, such as joint attention with a user or user gaze information. In this study, we propose the Gaze-grounded VQA dataset (GazeVQA) that clarifies ambiguous questions using gaze information by focusing on a clarification process complemented by gaze information. We also propose a method that utilizes gaze target estimation results to improve the accuracy of GazeVQA tasks. Our experimental results showed that the proposed method improved the performance in some cases of a VQA system on GazeVQA and identified some typical problems of GazeVQA tasks that need to be improved.

翻译：情境对话（如视觉问答VQA）常因依赖指示信息而产生歧义。在日语等语言中，由于常省略主语或宾语，这一问题更加突出。此类问题歧义通常可通过对话情境中的上下文（如用户共同注意或注视信息）得以澄清。本研究提出基于注视的VQA数据集（GazeVQA），通过聚焦注视信息辅助的澄清过程，利用注视信息消除歧义问题。我们还提出一种利用注视目标估计结果提升GazeVQA任务准确性的方法。实验结果表明，所提方法能在部分场景下提升GazeVQA系统中VQA模型的性能，同时揭示了GazeVQA任务中待改进的典型问题。