Skeleton sequence representation learning has shown great advantages for action recognition due to its promising ability to model human joints and topology. However, the current methods usually require sufficient labeled data for training computationally expensive models, which is labor-intensive and time-consuming. Moreover, these methods ignore how to utilize the fine-grained dependencies among different skeleton joints to pre-train an efficient skeleton sequence learning model that can generalize well across different datasets. In this paper, we propose an efficient skeleton sequence learning framework, named Skeleton Sequence Learning (SSL). To comprehensively capture the human pose and obtain discriminative skeleton sequence representation, we build an asymmetric graph-based encoder-decoder pre-training architecture named SkeletonMAE, which embeds skeleton joint sequence into Graph Convolutional Network (GCN) and reconstructs the masked skeleton joints and edges based on the prior human topology knowledge. Then, the pre-trained SkeletonMAE encoder is integrated with the Spatial-Temporal Representation Learning (STRL) module to build the SSL framework. Extensive experimental results show that our SSL generalizes well across different datasets and outperforms the state-of-the-art self-supervised skeleton-based action recognition methods on FineGym, Diving48, NTU 60 and NTU 120 datasets. Additionally, we obtain comparable performance to some fully supervised methods. The code is avaliable at https://github.com/HongYan1123/SkeletonMAE.

翻译：骨架序列表示学习因其对人体关节点及拓扑结构的出色建模能力，在动作识别领域展现出显著优势。然而，现有方法通常需要充足标注数据来训练计算成本高昂的模型，这一过程既耗时又费人力。此外，这些方法忽视了如何利用不同骨架关节点间的细粒度依赖关系，预训练一个能跨数据集泛化的高效骨架序列学习模型。本文提出一种高效骨架序列学习框架（Skeleton Sequence Learning，SSL）。为全面捕捉人体姿态并获取判别性骨架序列表示，我们构建了一种基于非对称图结构的编码器-解码器预训练架构SkeletonMAE，该架构将骨架关节点序列嵌入图卷积网络（Graph Convolutional Network，GCN），并基于先验人体拓扑知识重构被掩码的骨架关节点与边。随后，预训练的SkeletonMAE编码器与时空表示学习（Spatial-Temporal Representation Learning，STRL）模块集成，构建SSL框架。大量实验结果表明，我们的SSL框架在多个数据集上具有良好的泛化性能，在FineGym、Diving48、NTU 60和NTU 120数据集上均优于当前最先进的自监督骨架动作识别方法。此外，我们取得了与部分全监督方法相当的性能。代码已开源至https://github.com/HongYan1123/SkeletonMAE。