Multi-modal Large language models (MLLMs) show remarkable ability in video understanding. Nevertheless, understanding long videos remains challenging as the models can only process a finite number of frames in a single inference, potentially omitting crucial visual information. To address the challenge, we propose generating multiple predictions through visual context sampling, followed by a scoring mechanism to select the final prediction. Specifically, we devise a bin-wise sampling strategy that enables MLLMs to generate diverse answers based on various combinations of keyframes, thereby enriching the visual context. To determine the final prediction from the sampled answers, we employ a self-reward by linearly combining three scores: (1) a frequency score indicating the prevalence of each option, (2) a marginal confidence score reflecting the inter-intra sample certainty of MLLM predictions, and (3) a reasoning score for different question types, including clue-guided answering for global questions and temporal self-refocusing for local questions. The frequency score ensures robustness through majority correctness, the confidence-aligned score reflects prediction certainty, and the typed-reasoning score addresses cases with sparse key visual information using tailored strategies. Experiments show that this approach covers the correct answer for a high percentage of long video questions, on seven datasets show that our method improves the performance of three MLLMs.

翻译：多模态大语言模型（MLLMs）在视频理解方面展现出卓越能力。然而，理解长视频仍然具有挑战性，因为模型在单次推理中只能处理有限数量的帧，可能会遗漏关键的视觉信息。为应对这一挑战，我们提出通过视觉上下文采样生成多个预测，随后通过评分机制选择最终预测。具体而言，我们设计了一种分箱式采样策略，使MLLMs能够基于关键帧的不同组合生成多样化答案，从而丰富视觉上下文。为了从采样答案中确定最终预测，我们采用一种自奖励机制，该机制线性组合三个分数：(1) 频率分数，表示每个选项的普遍性；(2) 边际置信度分数，反映MLLM预测的样本间与样本内确定性；(3) 针对不同问题类型的推理分数，包括针对全局问题的线索引导回答和针对局部问题的时间自聚焦。频率分数通过多数正确性确保鲁棒性，置信度对齐分数反映预测确定性，而类型化推理分数则使用定制化策略处理关键视觉信息稀疏的情况。实验表明，该方法覆盖了高比例长视频问题的正确答案，在七个数据集上的实验显示，我们的方法提升了三种MLLMs的性能。