Video question--answering is a fundamental task in the field of video understanding. Although current vision--language models (VLMs) equipped with Video Transformers have enabled temporal modeling and yielded superior results, they are at the cost of huge computational power and thus too expensive to deploy in real-time application scenarios. An economical workaround only samples a small portion of frames to represent the main content of that video and tune an image--text model on these sampled frames. Recent video understanding models usually randomly sample a set of frames or clips, regardless of internal correlations between their visual contents, nor their relevance to the problem. We argue that such kinds of aimless sampling may omit the key frames from which the correct answer can be deduced, and the situation gets worse when the sampling sparsity increases, which always happens as the video lengths increase. To mitigate this issue, we propose two frame sampling strategies, namely the most domain frames (MDF) and most implied frames (MIF), to maximally preserve those frames that are most likely vital to the given questions. MDF passively minimizes the risk of key frame omission in a bootstrap manner, while MIS actively searches key frames customized for each video--question pair with the assistance of auxiliary models. The experimental results on three public datasets from three advanced VLMs (CLIP, GIT and All-in-one) demonstrate that our proposed strategies can boost the performance for image--text pretrained models. The source codes pertaining to the method proposed in this paper are publicly available at https://github.com/declare-lab/sas-vqa.

翻译：视频问答是视频理解领域中的一项基础任务。尽管当前配备视频Transformer的视觉-语言模型（VLM）已实现时间建模并获得了优越的结果，但这是以巨大的计算成本为代价的，因此难以在实时应用场景中部署。一种经济的替代方案是仅采样少量帧来表示视频的主要内容，并根据这些采样帧调整图像-文本模型。现有的视频理解模型通常随机采样一组帧或片段，而不考虑它们视觉内容之间的内在关联及其与问题的相关性。我们认为，此类无目的的采样可能会遗漏能够推导出正确答案的关键帧，并且随着视频长度的增加（采样稀疏性随之加剧），这一问题会愈加严重。为缓解此问题，我们提出了两种帧采样策略，即最相关领域帧（MDF）和最隐含帧（MIF），以最大限度地保留那些对给定问题最可能重要的帧。MDF以自举方式被动地降低关键帧遗漏的风险，而MIS则借助辅助模型主动搜索针对每个视频-问题对定制的关键帧。在基于三种先进VLM（CLIP、GIT和All-in-one）的三个公共数据集上的实验结果表明，我们提出的策略能够提升图像-文本预训练模型的性能。本文所提方法相关的源代码已在 https://github.com/declare-lab/sas-vqa 公开。