This paper proposes a novel multi-modal transformer network for detecting actions in untrimmed videos. To enrich the action features, our transformer network utilizes a new multi-modal attention mechanism that computes the correlations between different spatial and motion modalities combinations. Exploring such correlations for actions has not been attempted previously. To use the motion and spatial modality more effectively, we suggest an algorithm that corrects the motion distortion caused by camera movement. Such motion distortion, common in untrimmed videos, severely reduces the expressive power of motion features such as optical flow fields. Our proposed algorithm outperforms the state-of-the-art methods on two public benchmarks, THUMOS14 and ActivityNet. We also conducted comparative experiments on our new instructional activity dataset, including a large set of challenging classroom videos captured from elementary schools.

翻译：本文提出了一种新颖的多模态Transformer网络，用于检测未裁剪视频中的动作。为丰富动作特征，该Transformer网络采用了一种新的多模态注意力机制，能够计算不同空间模态与运动模态组合之间的相关性。此前尚未有研究探索过此类动作相关性。为更有效地利用运动模态与空间模态，我们提出了一种能够校正由摄像机运动引起的运动畸变算法。这种在未裁剪视频中常见的运动畸变会严重削弱光流场等运动特征的表达能力。所提算法在THUMOS14和ActivityNet两个公开基准上超越了现有最先进方法。此外，我们在新构建的教学活动数据集上进行了对比实验，该数据集包含从小学采集的大量具有挑战性的课堂视频。