Recent video recognition models utilize Transformer models for long-range spatio-temporal context modeling. Video transformer designs are based on self-attention that can model global context at a high computational cost. In comparison, convolutional designs for videos offer an efficient alternative but lack long-range dependency modeling. Towards achieving the best of both designs, this work proposes Video-FocalNet, an effective and efficient architecture for video recognition that models both local and global contexts. Video-FocalNet is based on a spatio-temporal focal modulation architecture that reverses the interaction and aggregation steps of self-attention for better efficiency. Further, the aggregation step and the interaction step are both implemented using efficient convolution and element-wise multiplication operations that are computationally less expensive than their self-attention counterparts on video representations. We extensively explore the design space of focal modulation-based spatio-temporal context modeling and demonstrate our parallel spatial and temporal encoding design to be the optimal choice. Video-FocalNets perform favorably well against the state-of-the-art transformer-based models for video recognition on five large-scale datasets (Kinetics-400, Kinetics-600, SS-v2, Diving-48, and ActivityNet-1.3) at a lower computational cost. Our code/models are released at https://github.com/TalalWasim/Video-FocalNets.

翻译：近期视频识别模型采用Transformer模型进行长程时空上下文建模。视频Transformer设计基于自注意力机制，虽能建模全局上下文但计算成本高昂。相比之下，针对视频的卷积设计虽提供了高效替代方案，却缺乏长程依赖建模能力。为兼顾两者优势，本文提出Video-FocalNet——一种兼具高效性与有效性的视频识别架构，可同时建模局部与全局上下文。Video-FocalNet基于时空焦点调制架构，通过反转自注意力机制中的交互与聚合步骤以提升效率。此外，聚合与交互步骤均采用计算开销低于视频表示中自注意力对应操作的卷积和逐元素乘法运算实现。我们全面探索了基于焦点调制的时空上下文建模设计空间，证明并行空间-时间编码设计为最优选择。在五个大规模数据集（Kinetics-400、Kinetics-600、SS-v2、Diving-48和ActivityNet-1.3）上，Video-FocalNet在更低计算成本下取得了优于现有最先进Transformer基视频识别模型的性能。我们的代码/模型已开源至https://github.com/TalalWasim/Video-FocalNets。