We introduce Mistake Attribution (MATT), a task for fine-grained understanding of human mistakes in egocentric video. Unlike prior mistake understanding work, which lacks fine-grained output, MATT concretely attributes mistakes to the input instruction text or the attempt video. MATT determines what part of the instruction is violated (semantic role), when the deviation becomes irreversible (the Point-of-No-Return, PNR), and where the mistake appears in the PNR frame. We develop MisEngine, a data engine that automatically constructs attribution-rich mistake samples from existing datasets and inherits their annotations. Applied to large egocentric corpora, MisEngine yields EPIC-KITCHENS-M and Ego4D-M, two datasets that are up to two orders of magnitude larger than prior mistake datasets. We then present MisFormer, a unified attention-based model for mistake attribution across semantic (what), temporal (when), and spatial (where) dimensions, trained using MisEngine supervision. Experiments on our new datasets and prior benchmarks show that MisFormer outperforms strong video-language, temporal localization, hand-object interaction, and mistake-detection baselines.

翻译：本文提出错误归因（MATT）任务，旨在对第一人称视频中的人为错误进行细粒度理解。与以往缺乏细粒度输出的错误理解研究不同，MATT将错误具体归因于输入指令文本或执行视频。该任务确定指令的哪部分被违反（语义角色）、偏差何时变得不可逆转（不可逆点，PNR），以及错误出现在PNR帧中的位置。我们开发了MisEngine数据引擎，能够从现有数据集中自动构建具有丰富归因信息的错误样本，并继承其标注。通过应用于大规模第一人称视频语料库，MisEngine生成了EPIC-KITCHENS-M和Ego4D-M两个数据集，其规模比现有错误数据集大两个数量级。进一步提出MisFormer模型——一种基于注意力的统一模型，可在语义（什么）、时间（何时）和空间（何处）维度上进行错误归因，并利用MisEngine的监督进行训练。在新数据集及现有基准测试上的实验表明，MisFormer在视频-语言理解、时序定位、手-物交互及错误检测等基线方法中均表现出优越性能。