Localizing people and recognizing their actions from videos is a challenging task towards high-level video understanding. Existing methods are mostly two-stage based, with one stage for person bounding box generation and the other stage for action recognition. However, such two-stage methods are generally with low efficiency. We observe that directly unifying detection and action recognition normally suffers from (i) inferior learning due to different desired properties of context representation for detection and action recognition; (ii) optimization difficulty with insufficient training data. In this work, we present a decoupled one-stage network dubbed DOAD, to mitigate above issues and improve the efficiency for spatio-temporal action detection. To achieve it, we decouple detection and action recognition into two branches. Specifically, one branch focuses on detection representation for actor detection, and the other one for action recognition. For the action branch, we design a transformer-based module (TransPC) to model pairwise relationships between people and context. Different from commonly used vector-based dot product in self-attention, it is built upon a novel matrix-based key and value for Hadamard attention to model person-context information. It not only exploits relationships between person pairs but also takes into account context and relative position information. The results on AVA and UCF101-24 datasets show that our method is competitive with two-stage state-of-the-art methods with significant efficiency improvement.

翻译：从视频中定位人物并识别其动作是一项面向高层视频理解的挑战性任务。现有方法大多基于两阶段架构，一阶段用于人物边界框生成，另一阶段用于动作识别。然而，这类两阶段方法通常效率较低。我们观察到，直接统一检测与动作识别通常面临以下问题：（i）由于检测与动作识别对上下文表征具有不同期望属性，导致学习效果不佳；（ii）在训练数据不足时存在优化困难。为此，本文提出一种名为DOAD的解耦式单阶段网络，以缓解上述问题并提升时空动作检测效率。具体而言，我们将检测与动作识别解耦为两个分支：一个分支专注于用于目标检测的检测表征，另一个分支专注于动作识别。针对动作分支，我们设计了一个基于Transformer的模块（TransPC）来建模人物与上下文之间的成对关系。与自注意力中常用的向量点积不同，该模块基于新颖的矩阵式键与值实现哈达玛注意力，以建模人物-上下文信息。它不仅能挖掘人物对之间的关联，还兼顾了上下文与相对位置信息。在AVA与UCF101-24数据集上的结果表明，本方法在显著提升效率的同时，达到了与两阶段最先进方法相媲美的性能。