Video understanding tasks have traditionally been modeled by two separate architectures, specially tailored for two distinct tasks. Sequence-based video tasks, such as action recognition, use a video backbone to directly extract spatiotemporal features, while frame-based video tasks, such as multiple object tracking (MOT), rely on single fixed-image backbone to extract spatial features. In contrast, we propose to unify video understanding tasks into one novel streaming video architecture, referred to as Streaming Vision Transformer (S-ViT). S-ViT first produces frame-level features with a memory-enabled temporally-aware spatial encoder to serve the frame-based video tasks. Then the frame features are input into a task-related temporal decoder to obtain spatiotemporal features for sequence-based tasks. The efficiency and efficacy of S-ViT is demonstrated by the state-of-the-art accuracy in the sequence-based action recognition task and the competitive advantage over conventional architecture in the frame-based MOT task. We believe that the concept of streaming video model and the implementation of S-ViT are solid steps towards a unified deep learning architecture for video understanding. Code will be available at https://github.com/yuzhms/Streaming-Video-Model.

翻译：视频理解任务传统上由两种专门针对不同任务设计的独立架构来建模。基于序列的视频任务（如动作识别）使用视频骨干网络直接提取时空特征，而基于帧的视频任务（如多目标跟踪）则依赖单一的固定图像骨干网络提取空间特征。相比之下，我们提出将视频理解任务统一到一种新颖的流式视频架构中，称为流式视觉Transformer（S-ViT）。S-ViT首先通过具备记忆能力的时间感知空间编码器生成帧级特征，以服务于基于帧的视频任务。然后将帧特征输入到与任务相关的时间解码器中，获取用于基于序列任务的时空特征。S-ViT在基于序列的动作识别任务中达到了最先进的准确率，并在基于帧的多目标跟踪任务中展现出优于传统架构的竞争力，从而证明了其高效性和有效性。我们相信，流式视频模型的概念以及S-ViT的实现，是迈向视频理解统一深度学习架构的坚实一步。代码将在https://github.com/yuzhms/Streaming-Video-Model 提供。