In this paper, we consider the problem of temporal action localization under low-shot (zero-shot & few-shot) scenario, with the goal of detecting and classifying the action instances from arbitrary categories within some untrimmed videos, even not seen at training time. We adopt a Transformer-based two-stage action localization architecture with class-agnostic action proposal, followed by open-vocabulary classification. We make the following contributions. First, to compensate image-text foundation models with temporal motions, we improve category-agnostic action proposal by explicitly aligning embeddings of optical flows, RGB and texts, which has largely been ignored in existing low-shot methods. Second, to improve open-vocabulary action classification, we construct classifiers with strong discriminative power, i.e., avoid lexical ambiguities. To be specific, we propose to prompt the pre-trained CLIP text encoder either with detailed action descriptions (acquired from large-scale language models), or visually-conditioned instance-specific prompt vectors. Third, we conduct thorough experiments and ablation studies on THUMOS14 and ActivityNet1.3, demonstrating the superior performance of our proposed model, outperforming existing state-of-the-art approaches by one significant margin.

翻译：本文研究了低样本（零样本与少样本）场景下的时序动作定位问题，目标是在未修剪视频中检测并分类任意类别（甚至训练时未见类别）的动作实例。我们采用基于Transformer的两阶段动作定位架构，包含类别无关的动作提议生成与开放词汇分类两个环节。本文贡献如下：第一，为弥补图像-文本基础模型在时序运动信息上的不足，我们通过显式对齐光流、RGB图像与文本的嵌入表示来改进类别无关的动作提议，这一点在现有低样本方法中常被忽视；第二，为提升开放词汇动作分类性能，我们构建具有强判别力的分类器，即避免词汇歧义。具体而言，我们提出对预训练CLIP文本编码器进行提示：既可利用大规模语言模型获取的详细动作描述，也可采用视觉条件化的实例特异性提示向量；第三，我们在THUMOS14和ActivityNet1.3数据集上进行了全面的实验与消融研究，结果表明本模型性能显著优于现有最优方法。