Current state-of-the-art video models process a video clip as a long sequence of spatio-temporal tokens. However, they do not explicitly model objects, their interactions across the video, and instead process all the tokens in the video. In this paper, we investigate how we can use knowledge of objects to design better video models, namely to process fewer tokens and to improve recognition accuracy. This is in contrast to prior works which either drop tokens at the cost of accuracy, or increase accuracy whilst also increasing the computation required. First, we propose an object-guided token sampling strategy that enables us to retain a small fraction of the input tokens with minimal impact on accuracy. And second, we propose an object-aware attention module that enriches our feature representation with object information and improves overall accuracy. Our resulting framework achieves better performance when using fewer tokens than strong baselines. In particular, we match our baseline with 30%, 40%, and 60% of the input tokens on SomethingElse, Something-something v2, and Epic-Kitchens, respectively. When we use our model to process the same number of tokens as our baseline, we improve by 0.6 to 4.2 points on these datasets.

翻译：当前最先进的视频模型将视频片段处理为长序列的时空令牌（spatio-temporal tokens）。然而，这些模型并未显式建模物体、物体在视频中的交互关系，而是直接处理视频中的所有令牌。本文旨在探究如何利用物体知识设计更优的视频模型，实现处理更少令牌的同时提升识别精度。不同于以往以牺牲精度为代价丢弃令牌、或虽提升精度却增加计算量的方法，我们首先提出物体引导的令牌采样策略，使得仅保留少量输入令牌也能使精度损失最小化；其次提出物体感知注意力模块，通过引入物体信息丰富特征表示，从而提升整体精度。所提出的框架在使用更少令牌时仍能取得优于强基线模型的表现——具体而言，在SomethingElse、Something-something v2和Epic-Kitchens数据集上，我们分别仅用30%、40%和60%的输入令牌即可达到基线的性能水平。当采用与基线相同数量的令牌时，本方法在这些数据集上的精度提升幅度为0.6至4.2个百分点。