Existing approaches to video understanding, mainly designed for short videos from a third-person perspective, are limited in their applicability in certain fields, such as robotics. In this paper, we delve into open-ended question-answering (QA) in long, egocentric videos, which allows individuals or robots to inquire about their own past visual experiences. This task presents unique challenges, including the complexity of temporally grounding queries within extensive video content, the high resource demands for precise data annotation, and the inherent difficulty of evaluating open-ended answers due to their ambiguous nature. Our proposed approach tackles these challenges by (i) integrating query grounding and answering within a unified model to reduce error propagation; (ii) employing large language models for efficient and scalable data synthesis; and (iii) introducing a close-ended QA task for evaluation, to manage answer ambiguity. Extensive experiments demonstrate the effectiveness of our method, which also achieves state-of-the-art performance on the QAEgo4D and Ego4D-NLQ benchmarks. Code, data, and models are available at https://github.com/Becomebright/GroundVQA.

翻译：现有的视频理解方法主要针对第三人称视角的短视频设计，在机器人等领域应用受限。本文探索了长时第一人称视频中的开放域问答（QA）任务，使个体或机器人能够查询自身过去的视觉经验。该任务面临独特挑战：在大量视频内容中实现查询的时间定位复杂度高、精确数据标注的资源需求高、以及开放域答案因模糊性而难以评估。我们提出的方法通过以下策略应对挑战：（i）将查询定位与答案生成整合于统一模型以减少误差传播；（ii）利用大语言模型实现高效可扩展的数据合成；（iii）引入封闭式QA任务进行答案歧义管理。大量实验证明了该方法的有效性，在QAEgo4D和Ego4D-NLQ基准上均达到最优性能。代码、数据和模型已开源至https://github.com/Becomebright/GroundVQA。