This paper presents a novel self-supervised temporal video alignment framework which is useful for several fine-grained human activity understanding applications. In contrast with the state-of-the-art method of CASA, where sequences of 3D skeleton coordinates are taken directly as input, our key idea is to use sequences of 2D skeleton heatmaps as input. Unlike CASA which performs self-attention in the temporal domain only, we feed 2D skeleton heatmaps to a video transformer which performs self-attention both in the spatial and temporal domains for extracting effective spatiotemporal and contextual features. In addition, we introduce simple heatmap augmentation techniques based on 2D skeletons for self-supervised learning. Despite the lack of 3D information, our approach achieves not only higher accuracy but also better robustness against missing and noisy keypoints than CASA. Furthermore, extensive evaluations on three public datasets, i.e., Penn Action, IKEA ASM, and H2O, demonstrate that our approach outperforms previous methods in different fine-grained human activity understanding tasks. Finally, fusing 2D skeleton heatmaps with RGB videos yields the state-of-the-art on all metrics and datasets. To the best of our knowledge, our work is the first to utilize 2D skeleton heatmap inputs and the first to explore multi-modality fusion for temporal video alignment.

翻译：本文提出了一种新颖的自监督时间视频对齐框架，该框架对多种细粒度人类活动理解应用具有重要价值。与当前最先进的CASA方法直接采用三维骨架坐标序列作为输入不同，我们的核心思想是使用二维骨架热图序列作为输入。CASA仅在时间域执行自注意力机制，而我们则将二维骨架热图输入视频变换器，该变换器在空间域和时间域同时执行自注意力机制，以提取有效的时空与上下文特征。此外，我们引入了基于二维骨架的简单热图增强技术用于自监督学习。尽管缺乏三维信息，我们的方法不仅比CASA实现了更高的精度，还在处理缺失和含噪关键点时表现出更强的鲁棒性。在Penn Action、IKEA ASM和H2O三个公开数据集上的广泛评估表明，我们的方法在不同细粒度人类活动理解任务中均优于现有方法。最终，将二维骨架热图与RGB视频融合在所有指标和数据集上达到了最先进水平。据我们所知，本文是首个采用二维骨架热图输入并探索多模态融合用于时间视频对齐的工作。