Reinforcement learning with verifiable rewards (RLVR) has advanced reasoning capabilities in multimodal large language models. However, existing methods typically treat visual inputs as deterministic, overlooking the perceptual ambiguity inherent to the visual modality. Consequently, they fail to distinguish whether a model's uncertainty stems from complex reasoning or ambiguous perception, preventing the targeted allocation of exploration or learning signals. To address this gap, we introduce DUPL, a dual-uncertainty guided policy learning approach for multimodal RLVR that quantifies and leverages both perceptual uncertainty (via symmetric KL divergence) and output uncertainty (via policy entropy) to guide policy updates. By establishing an uncertainty-driven feedback loop and employing a dynamic branch prioritization mechanism, DUPL recalibrates the policy advantage to focus learning on states with high perceptual or decisional ambiguity, enabling effective targeted exploration beyond passive data augmentation. Implemented on top of GRPO and evaluated on six multimodal mathematical and general-domain reasoning benchmarks, DUPL improves Qwen2.5-VL 3B and 7B models, achieving accuracy gains of up to 11.2% on visual math tasks and up to 7.1% on general-domain reasoning tasks, while consistently outperforming GRPO. These results demonstrate that dual-uncertainty guided policy learning is an effective and generalizable approach for multimodal RLVR.

翻译：基于可验证奖励的强化学习（RLVR）提升了多模态大语言模型的推理能力。然而，现有方法通常将视觉输入视为确定性的，忽视了视觉模态固有的感知模糊性。因此，它们无法区分模型的不确定性是源于复杂的推理过程还是模糊的感知输入，从而无法有针对性地分配探索或学习信号。为弥补这一不足，我们提出了DUPL，一种用于多模态RLVR的双重不确定性引导策略学习方法。该方法量化并利用感知不确定性（通过对称KL散度）和输出不确定性（通过策略熵）来指导策略更新。通过建立一个不确定性驱动的反馈循环并采用动态分支优先级机制，DUPL重新校准了策略优势，将学习重点集中在具有高感知或决策模糊性的状态上，从而实现了超越被动数据增强的有效针对性探索。DUPL在GRPO基础上实现，并在六个多模态数学及通用领域推理基准上进行了评估。实验表明，DUPL提升了Qwen2.5-VL 3B和7B模型的性能，在视觉数学任务上准确率最高提升11.2%，在通用领域推理任务上最高提升7.1%，且始终优于GRPO。这些结果表明，双重不确定性引导的策略学习是一种有效且可推广的多模态RLVR方法。