Given a video with $T$ frames, frame sampling is a task to select $N \ll T$ frames, so as to maximize the performance of a fixed video classifier. Not just brute-force search, but most existing methods suffer from its vast search space of $\binom{T}{N}$, especially when $N$ gets large. To address this challenge, we introduce a novel perspective of reducing the search space from $O(T^N)$ to $O(T)$. Instead of exploring the entire $O(T^N)$ space, our proposed semi-optimal policy selects the top $N$ frames based on the independently estimated value of each frame using per-frame confidence, significantly reducing the computational complexity. We verify that our semi-optimal policy can efficiently approximate the optimal policy, particularly under practical settings. Additionally, through extensive experiments on various datasets and model architectures, we demonstrate that learning our semi-optimal policy ensures stable and high performance regardless of the size of $N$ and $T$.

翻译：给定一个包含 $T$ 帧的视频，帧采样任务旨在选择 $N \ll T$ 帧，以最大化固定视频分类器的性能。不仅是暴力搜索，大多数现有方法都受困于其 $\binom{T}{N}$ 的巨大搜索空间，尤其是在 $N$ 较大时。为应对这一挑战，我们引入了一种将搜索空间从 $O(T^N)$ 缩减至 $O(T)$ 的新视角。我们提出的半最优策略并非探索整个 $O(T^N)$ 空间，而是基于使用逐帧置信度独立估计的每帧价值，选择价值最高的 $N$ 帧，从而显著降低了计算复杂度。我们验证了该半最优策略能够高效地逼近最优策略，尤其是在实际应用场景下。此外，通过在多种数据集和模型架构上进行大量实验，我们证明学习该半最优策略能够确保稳定且高性能的表现，而不受 $N$ 和 $T$ 大小的影响。