While reinforcement learning (RL) over chains of thought has significantly advanced language models in tasks such as mathematics and coding, visual reasoning introduces added complexity by requiring models to direct visual attention, interpret perceptual inputs, and ground abstract reasoning in spatial evidence. We introduce ViGoRL (Visually Grounded Reinforcement Learning), a vision-language model trained with RL to explicitly anchor each reasoning step to specific visual coordinates. Inspired by human visual decision-making, ViGoRL learns to produce spatially grounded reasoning traces, guiding visual attention to task-relevant regions at each step. When fine-grained exploration is required, our novel multi-turn RL framework enables the model to dynamically zoom into predicted coordinates as reasoning unfolds. Across a diverse set of visual reasoning benchmarks--including SAT-2 and BLINK for spatial reasoning, V*bench for visual search, and ScreenSpot and VisualWebArena for web-based grounding--ViGoRL consistently outperforms both supervised fine-tuning and conventional RL baselines that lack explicit grounding mechanisms. Incorporating multi-turn RL with zoomed-in visual feedback significantly improves ViGoRL's performance on localizing small GUI elements and visual search, achieving 86.4% on V*Bench. Additionally, we find that grounding amplifies other visual behaviors such as region exploration, grounded subgoal setting, and visual verification. Finally, human evaluations show that the model's visual references are not only spatially accurate but also helpful for understanding model reasoning steps. Our results show that visually grounded RL is a strong paradigm for imbuing models with general-purpose visual reasoning.

翻译：尽管基于思维链的强化学习在数学和编程等任务中显著提升了语言模型的能力，但视觉推理引入了额外复杂性，要求模型定向视觉注意、解读感知输入，并将抽象推理锚定于空间证据。我们提出ViGoRL（视觉具身强化学习），这是一种通过强化学习训练的视觉语言模型，能够将每个推理步骤明确锚定到具体的视觉坐标上。受人类视觉决策机制的启发，ViGoRL学习生成空间具身的推理轨迹，逐步引导视觉注意聚焦于任务相关区域。当需要细粒度探索时，我们提出的多轮强化学习框架使模型能够在推理过程中动态放大到预测坐标。在涵盖SPAT-2与BLINK空间推理、V*Bench视觉搜索、ScreenSpot与VisualWebArena网页界面定位等多样化视觉推理基准测试中，ViGoRL始终优于缺乏显式定位机制的有监督微调和传统强化学习基线。结合放大的视觉反馈的多轮强化学习显著提升了ViGoRL在定位小型GUI元素和视觉搜索中的性能，在V*Bench上达到86.4%。此外，我们发现定位能力放大了区域探索、具身子目标设定和视觉验证等其他视觉行为。最后，人工评估表明模型生成的视觉参考不仅空间精度高，且有助于理解推理步骤。实验结果表明，视觉具身强化学习是赋予模型通用视觉推理能力的有效范式。