Semi-supervised temporal action segmentation (SS-TA) aims to perform frame-wise classification in long untrimmed videos, where only a fraction of videos in the training set have labels. Recent studies have shown the potential of contrastive learning in unsupervised representation learning using unlabelled data. However, learning the representation of each frame by unsupervised contrastive learning for action segmentation remains an open and challenging problem. In this paper, we propose a novel Semantic-guided Multi-level Contrast scheme with a Neighbourhood-Consistency-Aware unit (SMC-NCA) to extract strong frame-wise representations for SS-TAS. Specifically, for representation learning, SMC is first used to explore intra- and inter-information variations in a unified and contrastive way, based on action-specific semantic information and temporal information highlighting relations between actions. Then, the NCA module, which is responsible for enforcing spatial consistency between neighbourhoods centered at different frames to alleviate over-segmentation issues, works alongside SMC for semi-supervised learning (SSL). Our SMC outperforms the other state-of-the-art methods on three benchmarks, offering improvements of up to 17.8% and 12.6% in terms of Edit distance and accuracy, respectively. Additionally, the NCA unit results in significantly better segmentation performance in the presence of only 5% labelled videos. We also demonstrate the generalizability and effectiveness of the proposed method on our Parkinson Disease's Mouse Behaviour (PDMB) dataset. Code is available at https://github.com/FeixiangZhou/SMC-NCA.

翻译：半监督时序动作分割（SS-TAS）旨在对长未剪辑视频进行逐帧分类，其中训练集中仅部分视频带有标注。近期研究表明，利用未标注数据进行无监督对比学习在表示学习方面具有潜力。然而，通过无监督对比学习为动作分割任务学习逐帧表示仍是一个开放且具有挑战性的问题。本文提出一种新颖的语义引导多层级对比方案，结合邻域一致性感知单元（SMC-NCA），为SS-TAS提取强判别性的逐帧表示。具体而言，在表示学习阶段，首先基于动作特定的语义信息以及强调动作间关系的时序信息，采用SMC方案以统一且对比的方式探索帧内与帧间信息的变化。随后，NCA模块负责增强以不同帧为中心的邻域间的空间一致性，以缓解过分割问题，并与SMC协同工作实现半监督学习（SSL）。我们的SMC方案在三个基准数据集上均优于其他最先进方法，在编辑距离和准确率指标上分别实现了最高17.8%和12.6%的提升。此外，在仅使用5%标注视频的情况下，NCA单元仍能显著提升分割性能。我们还在自建的帕金森病小鼠行为（PDMB）数据集上验证了所提方法的泛化能力与有效性。代码发布于https://github.com/FeixiangZhou/SMC-NCA。