Large-scale video-language pre-training has made remarkable strides in advancing video-language understanding tasks. However, the heavy computational burden of video encoding remains a formidable efficiency bottleneck, particularly for long-form videos. These videos contain massive visual tokens due to their inherent 3D properties and spatiotemporal redundancy, making it challenging to capture complex temporal and spatial relationships. To tackle this issue, we propose an efficient method called TEmporal-Spatial Token Aggregation (TESTA). TESTA condenses video semantics by adaptively aggregating similar frames, as well as similar patches within each frame. TESTA can reduce the number of visual tokens by 75% and thus accelerate video encoding. Building upon TESTA, we introduce a pre-trained video-language model equipped with a divided space-time token aggregation module in each video encoder block. We evaluate our model on five datasets for paragraph-to-video retrieval and long-form VideoQA tasks. Experimental results show that TESTA improves computing efficiency by 1.7 times, and achieves significant performance gains from its scalability in processing longer input frames, e.g., +13.7 R@1 on QuerYD and +6.5 R@1 on Condensed Movie.

翻译：大规模视频语言预训练在推动视频语言理解任务方面取得了显著进展。然而，视频编码的巨大计算负担仍是关键的效率瓶颈，尤其对于长视频而言。这些视频因固有的三维属性和时空冗余而包含海量视觉标记，使得捕获复杂的时空关系极具挑战。为解决该问题，我们提出一种名为时空标记聚合（TESTA）的高效方法。TESTA通过自适应聚合相似帧及帧内相似图像块来压缩视频语义。该方法可将视觉标记数量减少75%，从而加速视频编码。基于TESTA，我们引入一个预训练视频语言模型，该模型在每个视频编码器块中配备分立的时空标记聚合模块。我们在五个数据集上对段落级视频检索和长视频问答任务进行评估。实验结果表明，TESTA将计算效率提升1.7倍，且因其处理更长输入帧的可扩展性而获得显著性能提升，例如在QuerYD数据集中R@1提升13.7个百分点，在Condensed Movie数据集中R@1提升6.5个百分点。