We present Egocentric Action Scene Graphs (EASGs), a new representation for long-form understanding of egocentric videos. EASGs extend standard manually-annotated representations of egocentric videos, such as verb-noun action labels, by providing a temporally evolving graph-based description of the actions performed by the camera wearer, including interacted objects, their relationships, and how actions unfold in time. Through a novel annotation procedure, we extend the Ego4D dataset by adding manually labeled Egocentric Action Scene Graphs offering a rich set of annotations designed for long-from egocentric video understanding. We hence define the EASG generation task and provide a baseline approach, establishing preliminary benchmarks. Experiments on two downstream tasks, egocentric action anticipation and egocentric activity summarization, highlight the effectiveness of EASGs for long-form egocentric video understanding. We will release the dataset and the code to replicate experiments and annotations.

翻译：我们提出了第一人称动作场景图（Egocentric Action Scene Graphs，EASGs），这是一种用于第一人称视频长程理解的新型表示方法。EASG通过提供基于图的、随时间演进的描述，刻画了佩戴相机者执行的动作（包括交互对象、对象间关系以及动作如何随时间展开），从而扩展了标准的人工标注第一人称视频表示形式（如动宾式动作标签）。通过一种新颖的标注流程，我们为Ego4D数据集增添了人工标注的第一人称动作场景图，提供了一套丰富的标注信息，专用于长程第一人称视频理解。据此，我们定义了EASG生成任务，并提供了基线方法，建立了初步基准。针对两项下游任务（第一人称动作预测与第一人称活动摘要生成）的实验，凸显了EASG在长程第一人称视频理解中的有效性。我们将发布数据集及用于复现实验与标注的代码。