Efficient video-language modeling should consider the computational cost because of a large, sometimes intractable, number of video frames. Parametric approaches such as the attention mechanism may not be ideal since its computational cost quadratically increases as the video length increases. Rather, previous studies have relied on offline feature extraction or frame sampling to represent the video efficiently, focusing on cross-modal modeling in short video clips. In this paper, we propose a semi-parametric video-grounded text generation model, SeViT, a novel perspective on scalable video-language modeling toward long untrimmed videos. Treating a video as an external data store, SeViT includes a non-parametric frame retriever to select a few query-relevant frames from the data store for a given query and a parametric generator to effectively aggregate the frames with the query via late fusion methods. Experimental results demonstrate our method has a significant advantage in longer videos and causal video understanding. Moreover, our model achieves the new state of the art on four video-language datasets, iVQA (+4.8), Next-QA (+6.9), and Activitynet-QA (+4.8) in accuracy, and MSRVTT-Caption (+3.6) in CIDEr.

翻译：高效的视频-语言建模应考虑计算成本，这是因为视频帧数庞大，有时甚至难以处理。诸如注意力机制等参数化方法可能并不理想，因为其计算成本随视频长度呈二次方增长。相反，先前的研究依赖于离线特征提取或帧采样来高效表示视频，侧重于短片段中的跨模态建模。本文提出一种半参数化视频驱动文本生成模型SeViT，为面向长未裁剪视频的可扩展视频-语言建模提供了全新视角。通过将视频视为外部数据存储，SeViT包含一个非参数化帧检索器，用于从数据存储中为给定查询选择少量查询相关帧；以及一个参数化生成器，通过后期融合方法有效聚合查询与帧。实验结果表明，我们的方法在处理更长视频及因果视频理解方面具有显著优势。此外，模型在四个视频-语言数据集上实现了最新最优性能：在iVQA（+4.8）、Next-QA（+6.9）和Activitynet-QA（+4.8）上准确率提升，在MSRVTT-Caption（+3.6）上CIDEr指标提升。