Skeleton-based action segmentation requires recognizing composable actions in untrimmed videos. Current approaches decouple this problem by first extracting local visual features from skeleton sequences and then processing them by a temporal model to classify frame-wise actions. However, their performances remain limited as the visual features cannot sufficiently express composable actions. In this context, we propose Latent Action Composition (LAC), a novel self-supervised framework aiming at learning from synthesized composable motions for skeleton-based action segmentation. LAC is composed of a novel generation module towards synthesizing new sequences. Specifically, we design a linear latent space in the generator to represent primitive motion. New composed motions can be synthesized by simply performing arithmetic operations on latent representations of multiple input skeleton sequences. LAC leverages such synthesized sequences, which have large diversity and complexity, for learning visual representations of skeletons in both sequence and frame spaces via contrastive learning. The resulting visual encoder has a high expressive power and can be effectively transferred onto action segmentation tasks by end-to-end fine-tuning without the need for additional temporal models. We conduct a study focusing on transfer-learning and we show that representations learned from pre-trained LAC outperform the state-of-the-art by a large margin on TSU, Charades, PKU-MMD datasets.

翻译：基于骨骼的动作分割需要识别未修剪视频中的可组合动作。当前方法将这一问题解耦，首先从骨骼序列中提取局部视觉特征，然后通过时间模型进行分类以逐帧标注动作。然而，由于视觉特征无法充分表达可组合动作，其性能仍受限。在此背景下，我们提出潜在动作组合（LAC），一种新型自监督框架，旨在通过学习合成的可组合运动来实现基于骨骼的动作分割。LAC包含一个用于合成新序列的新型生成模块。具体而言，我们在生成器中设计了一个线性潜在空间来表示原始运动。通过对多个输入骨骼序列的潜在表示进行简单的算术运算，即可合成新的组合运动。LAC利用这些具有高度多样性和复杂性的合成序列，通过对比学习在序列和帧空间中学习骨骼的视觉表征。由此获得的视觉编码器具有高表达能力，可通过端到端微调有效迁移到动作分割任务中，无需额外的时间模型。我们专注于迁移学习的研究表明，从预训练LAC中学习的表征在TSU、Charades、PKU-MMD数据集上大幅超越了当前最优方法。