Vision-Language Models (VLMs) often suffer from visual hallucinations: generating things that are not consistent with visual inputs and language shortcuts, where they skip the visual part and just rely on text priors. These issues arise because most post training methods for VLMs rely on simple verifiable answer matching and supervise only final outputs, leaving intermediate visual reasoning without explicit guidance. As a result, VLMs receive sparse visual signals and often learn to prioritize language based reasoning over visual perception. We introduce Vision SR1, a three stage self rewarding reinforcement learning method that improves visual reasoning without relying on external visual supervision. Vision SR1 decomposes VLM reasoning into two components: visual reasoning and language reasoning, where the model is first prompted to produce self-contained visual descriptions sufficient to answer the question without referring back to the input image, before jointly optimizing both visual and language reasoning through our multi reward loss objective. To validate this self containment, the same VLM model is reprompted to perform language reasoning using only the generated visual reasoning as input to compute visual reward. The final reward is computed through a decoupled reward-advantage framework, where visual reward and language reasoning reward each have their advantages calculated separately. Our experiments show that Vision SR1 improves visual reasoning, mitigates visual hallucinations, and reduces reliance on language shortcuts across diverse vision language tasks, while being more efficient than methods that rely on external visual reward models, which require additional GPUs to host. In contrast, Vision SR1 introduces no extra GPU overhead beyond that of standard training.

翻译：视觉语言模型常出现视觉幻觉：生成与视觉输入不一致的内容，以及语言捷径现象——跳过视觉部分仅依赖文本先验。这些问题源于大多数视觉语言模型的后训练方法仅依赖简单的可验证答案匹配，仅对最终输出进行监督，导致中间视觉推理缺乏明确引导。因此，视觉语言模型获得稀疏的视觉信号，常常倾向于优先采用基于语言的推理而非视觉感知。我们提出Vision SR1，一种三阶段自奖励强化学习方法，可在无需外部视觉监督的情况下改进视觉推理。Vision SR1将视觉语言模型的推理分解为两个组件：视觉推理和语言推理。模型首先被提示生成自包含的视觉描述——该描述需足以在无需参考输入图像的情况下回答问题，随后通过我们的多奖励损失目标联合优化视觉和语言推理。为验证这种自包含性，我们重新提示同一视觉语言模型，仅使用生成的视觉推理作为输入执行语言推理以计算视觉奖励。最终奖励通过解耦的奖励-优势框架计算，其中视觉奖励和语言推理奖励分别计算各自的优势。实验表明，Vision SR1在多种视觉语言任务上改进了视觉推理，缓解了视觉幻觉，并减少了对语言捷径的依赖，同时比依赖外部视觉奖励模型（需额外GPU支持）的方法更高效。相比之下，Vision SR1在标准训练基础上未引入额外GPU开销。