Zero-shot action recognition (ZSAR) aims to learn an alignment model between videos and class descriptions of seen actions that is transferable to unseen actions. The text queries (class descriptions) used in existing ZSAR works, however, are often short action names that fail to capture the rich semantics in the videos, leading to misalignment. With the intuition that video content descriptions (e.g., video captions) can provide rich contextual information of visual concepts in videos, we propose to utilize human annotated video descriptions to enrich the semantics of the class descriptions of each action. However, all existing action video description datasets are limited in terms of the number of actions, the semantics of video descriptions, etc. To this end, we collect a large-scale action video descriptions dataset named ActionHub, which covers a total of 1,211 common actions and provides 3.6 million action video descriptions. With the proposed ActionHub dataset, we further propose a novel Cross-modality and Cross-action Modeling (CoCo) framework for ZSAR, which consists of a Dual Cross-modality Alignment module and a Cross-action Invariance Mining module. Specifically, the Dual Cross-modality Alignment module utilizes both action labels and video descriptions from ActionHub to obtain rich class semantic features for feature alignment. The Cross-action Invariance Mining module exploits a cycle-reconstruction process between the class semantic feature spaces of seen actions and unseen actions, aiming to guide the model to learn cross-action invariant representations. Extensive experimental results demonstrate that our CoCo framework significantly outperforms the state-of-the-art on three popular ZSAR benchmarks (i.e., Kinetics-ZSAR, UCF101 and HMDB51) under two different learning protocols in ZSAR. We will release our code, models, and the proposed ActionHub dataset.

翻译：零样本动作识别（ZSAR）旨在学习视频与可见动作类别描述之间的对齐模型，并将该模型迁移至不可见动作。然而，现有ZSAR工作中使用的文本查询（类别描述）通常为简短的动作名称，难以捕捉视频中的丰富语义，导致对齐偏差。基于视频内容描述（如视频字幕）能提供视觉概念丰富上下文信息的直觉，我们提出利用人工标注的视频描述来增强每个动作类别描述的语义。但现有动作视频描述数据集在动作数量、视频描述语义丰富性等方面均存在局限。为此，我们构建了名为ActionHub的大规模动作视频描述数据集，涵盖1,211个常见动作，并提供360万条动作视频描述。基于所提出的ActionHub数据集，我们进一步提出面向ZSAR的跨模态与跨动作建模（CoCo）框架，该框架包含双跨模态对齐模块和跨动作不变性挖掘模块。具体而言，双跨模态对齐模块通过ActionHub中的动作标签和视频描述获取丰富的类别语义特征以进行特征对齐；跨动作不变性挖掘模块利用可见动作与不可见动作类别语义特征空间之间的循环重建过程，引导模型学习跨动作不变性表示。大量实验结果表明，在ZSAR的两种不同学习协议下，我们的CoCo框架在三个主流ZSAR基准（即Kinetics-ZSAR、UCF101和HMDB51）上均显著优于现有最先进方法。我们将公开代码、模型以及所提出的ActionHub数据集。