Generalized zero-shot skeleton-based action recognition (GZSSAR) is a new challenging problem in computer vision community, which requires models to recognize actions without any training samples. Previous studies only utilize the action labels of verb phrases as the semantic prototypes for learning the mapping from skeleton-based actions to a shared semantic space. However, the limited semantic information of action labels restricts the generalization ability of skeleton features for recognizing unseen actions. In order to solve this dilemma, we propose a multi-semantic fusion (MSF) model for improving the performance of GZSSAR, where two kinds of class-level textual descriptions (i.e., action descriptions and motion descriptions), are collected as auxiliary semantic information to enhance the learning efficacy of generalizable skeleton features. Specially, a pre-trained language encoder takes the action descriptions, motion descriptions and original class labels as inputs to obtain rich semantic features for each action class, while a skeleton encoder is implemented to extract skeleton features. Then, a variational autoencoder (VAE) based generative module is performed to learn a cross-modal alignment between skeleton and semantic features. Finally, a classification module is built to recognize the action categories of input samples, where a seen-unseen classification gate is adopted to predict whether the sample comes from seen action classes or not in GZSSAR. The superior performance in comparisons with previous models validates the effectiveness of the proposed MSF model on GZSSAR.

翻译：广义零样本骨架动作识别（GZSSAR）是计算机视觉领域一项新的挑战性问题，要求模型在没有任何训练样本的情况下识别动作。以往研究仅利用动词短语形式的动作标签作为语义原型，用于学习从骨架动作到共享语义空间的映射。然而，动作标签有限的语义信息限制了骨架特征对未见动作的泛化能力。为解决这一困境，我们提出一种多语义融合（MSF）模型以提升GZSSAR性能，该模型收集两类类别级文本描述（即动作描述和运动描述）作为辅助语义信息，以增强可泛化骨架特征的学习效果。具体而言，预训练语言编码器将动作描述、运动描述和原始类别标签作为输入，为每个动作类别获取丰富的语义特征；同时，骨架编码器用于提取骨架特征。随后，基于变分自编码器（VAE）的生成模块被用于学习骨架特征与语义特征之间的跨模态对齐。最后，构建分类模块以识别输入样本的动作类别，其中采用可见-不可见分类门控预测样本是否来自已知动作类别，以应对GZSSAR场景。与以往模型的对比实验表明，所提出的MSF模型在GZSSAR任务上具有卓越性能。