We introduce CPPO, a Contrastive Perception Policy Optimization method for finetuning vision-language models (VLMs). While reinforcement learning (RL) has advanced reasoning in language models, extending it to multimodal reasoning requires improving both the perception and reasoning aspects. Prior works tackle this challenge mainly with explicit perception rewards, but disentangling perception tokens from reasoning tokens is difficult, requiring extra LLMs, ground-truth data, forced separation of perception from reasoning by policy model, or applying rewards indiscriminately to all output tokens. CPPO addresses this problem by detecting perception tokens via entropy shifts in the model outputs under perturbed input images. CPPO then extends the RL objective function with a Contrastive Perception Loss (CPL) that enforces consistency under information-preserving perturbations and sensitivity under information-removing ones. Experiments show that CPPO surpasses previous perception-rewarding methods, while avoiding extra models, making training more efficient and scalable.

翻译：本文提出CPPO（对比感知策略优化方法），用于微调视觉语言模型（VLM）。尽管强化学习（RL）已提升了语言模型的推理能力，但将其扩展至多模态推理需要同时改进感知与推理两方面。先前研究主要通过显式感知奖励应对这一挑战，但将感知标记与推理标记分离存在困难，往往需要额外的大型语言模型、真实标注数据、强制策略模型分离感知与推理，或对所有输出标记不加区分地施加奖励。CPPO通过检测输入图像受扰动时模型输出的熵值变化来识别感知标记，进而扩展RL目标函数，引入对比感知损失（CPL）。该损失函数强制模型在信息保持型扰动下保持一致性，在信息消除型扰动下保持敏感性。实验表明，CPPO在超越先前感知奖励方法的同时，无需引入额外模型，使训练更高效且可扩展。