Video question answering (VQA) with vision-language models (VLMs) depends critically on which frames are selected from the input video, yet most systems rely on uniform or heuristic sampling that cannot be optimized for downstream answering quality. We introduce \textbf{HORNet}, a lightweight frame selection policy trained with Group Relative Policy Optimization (GRPO) to learn which frames a frozen VLM needs to answer questions correctly. With fewer than 1M trainable parameters, HORNet reduces input frames by up to 99\% and VLM processing time by up to 93\%, while improving answer quality on short-form benchmarks (+1.7\% F1 on MSVD-QA) and achieving strong performance on temporal reasoning tasks (+7.3 points over uniform sampling on NExT-QA). We formalize this as Select Any Frames (SAF), a task that decouples visual input curation from VLM reasoning, and show that GRPO-trained selection generalizes better out-of-distribution than supervised and PPO alternatives. HORNet's policy further transfers across VLM answerers without retraining, yielding an additional 8.5\% relative gain when paired with a stronger model. Evaluated across six benchmarks spanning 341,877 QA pairs and 114.2 hours of video, our results demonstrate that optimizing \emph{what} a VLM sees is a practical and complementary alternative to optimizing what it generates while improving efficiency. Code is available at https://github.com/ostadabbas/HORNet.

翻译：视频问答（VQA）与视觉语言模型（VLM）的性能关键取决于从输入视频中选取哪些帧，但现有系统多采用均匀或启发式采样策略，无法针对下游答案质量进行优化。我们提出**HORNet**，一种轻量级帧选择策略，通过组相对策略优化（GRPO）训练，学习冻结型VLM需要哪些帧才能正确回答问题。以不足100万个可训练参数实现的HORNet，可将输入帧减少99%，VLM处理时间缩减93%，同时在短视频基准测试中提升答案质量（MSVD-QA的F1值提高1.7%），并在时序推理任务中取得优异表现（NExT-QA相比均匀采样提升7.3个点）。我们将此形式化为“任意帧选择”（SAF）任务——将视觉输入筛选与VLM推理解耦，并证明GRPO训练的帧选择在分布外泛化能力上优于监督学习和PPO替代方案。HORNet的策略还可跨VLM答案器迁移而无需重新训练，当与更强模型配合时额外实现8.5%的相对增益。在覆盖341,877个问答对和114.2小时视频的六个基准测试中，我们的结果表明：优化VLM“看到什么”是优化其生成内容的实用且互补的替代方案，同时提升效率。代码已开源：https://github.com/ostadabbas/HORNet