3D Skeleton-based human action recognition has attracted increasing attention in recent years. Most of the existing work focuses on supervised learning which requires a large number of labeled action sequences that are often expensive and time-consuming to annotate. In this paper, we address self-supervised 3D action representation learning for skeleton-based action recognition. We investigate self-supervised representation learning and design a novel skeleton cloud colorization technique that is capable of learning spatial and temporal skeleton representations from unlabeled skeleton sequence data. We represent a skeleton action sequence as a 3D skeleton cloud and colorize each point in the cloud according to its temporal and spatial orders in the original (unannotated) skeleton sequence. Leveraging the colorized skeleton point cloud, we design an auto-encoder framework that can learn spatial-temporal features from the artificial color labels of skeleton joints effectively. Specifically, we design a two-steam pretraining network that leverages fine-grained and coarse-grained colorization to learn multi-scale spatial-temporal features. In addition, we design a Masked Skeleton Cloud Repainting task that can pretrain the designed auto-encoder framework to learn informative representations. We evaluate our skeleton cloud colorization approach with linear classifiers trained under different configurations, including unsupervised, semi-supervised, fully-supervised, and transfer learning settings. Extensive experiments on NTU RGB+D, NTU RGB+D 120, PKU-MMD, NW-UCLA, and UWA3D datasets show that the proposed method outperforms existing unsupervised and semi-supervised 3D action recognition methods by large margins and achieves competitive performance in supervised 3D action recognition as well.

翻译：三维骨架人体动作识别近年来受到越来越多的关注。现有研究主要聚焦于监督学习，但该方法需要大量标注动作序列，而标注工作往往昂贵且耗时。本文针对基于骨架的动作识别任务，研究自监督三维动作表示学习。我们探索自监督表示学习技术，设计了一种新颖的骨架云着色方法，能够从未标注的骨架序列数据中学习时空骨架表示。我们将动作骨架序列表示为三维骨架云，并根据原始（未标注）骨架序列中的时间与空间顺序对云中每个点进行着色。利用被着色的骨架点云，我们构建了一个自编码器框架，该框架可有效从骨架关节的人工颜色标签中学习时空特征。具体而言，我们设计了一个双通道预训练网络，通过细粒度与粗粒度着色相结合的方式学习多尺度时空特征。此外，我们还提出掩码骨架云重绘任务，用于预训练所设计的自编码器框架，从而学习信息丰富的表示。我们在不同配置下（包括无监督、半监督、全监督和迁移学习设置）训练线性分类器，以评估骨架云着色方法。在NTU RGB+D、NTU RGB+D 120、PKU-MMD、NW-UCLA和UWA3D数据集上的大量实验表明，所提出的方法在无监督和半监督三维动作识别任务中显著优于现有方法，同时在监督三维动作识别中也取得了具有竞争力的表现。