Vision-Language Models (VLMs) exhibit strong performance in instruction following and open-ended vision-language reasoning, yet they frequently generate fluent outputs that are weakly grounded in visual evidence. Prior works have shown that instruction prompting further worsens this issue by amplifying language priors, especially when the visual signal is uncertain or ambiguous. To address this challenge, we propose a decoding framework that explicitly balances linguistic informativeness and visual faithfulness during generation. Our method, Instruction-Evidence Contrastive Dual-Stream Decoding (IECD2), maintains two parallel probability distributions of tokens at each decoding step: an instruction-driven stream that promotes expressive and informative responses, and an evidence-driven stream that enforces strict grounding in the image. These two streams are adaptively fused using a symmetric KL-based contrast-based gate, which suppresses tokens favored by language priors but unsupported by visual evidence, while preserving them when both distributions agree. We evaluate IECD2 on multiple datasets spanning various generative vision-language reasoning tasks such as captioning and visual question answering, including POPE, MME, VQAv2, AMBER, MS-COCO, and LLaVA-Bench. IECD2 demonstrates consistent improvements in task accuracy and reasoning performance, alongside a substantial reduction in hallucination across all evaluation metrics compared to state-of-the-art decoding approaches.

翻译：视觉-语言模型在指令遵循和开放式视觉-语言推理中表现出强劲性能，但常生成流畅却缺乏视觉证据支撑的输出。先前研究表明，指令提示通过放大语言先验进一步加剧此问题，尤其在视觉信号不确定或模糊时。为解决这一挑战，我们提出一种在生成过程中明确平衡语言信息量与视觉忠实度的解码框架。我们的方法——指令-证据对比双流解码（IECD2）——在每个解码步骤维护两个并行的token概率分布：指令驱动流促进表达性和信息性响应，证据驱动流强制严格锚定图像内容。通过对称KL对比门控机制自适应融合这两条流，该门控能抑制受语言先验青睐但缺乏视觉证据支持的token，同时在两分布一致时保留它们。我们在涵盖字幕生成和视觉问答等多项生成性视觉-语言推理任务的数据集（包括POPE、MME、VQAv2、AMBER、MS-COCO和LLaVA-Bench）上评估IECD2。与最先进的解码方法相比，IECD2在任务准确率和推理性能上持续提升，同时所有评估指标中的幻觉现象均大幅减少。