Vision-Language Models (VLMs) hallucinate objects that are not present, and a growing line of work tries to curb this by feeding the model its own generated caption as auxiliary evidence -- assuming that a caption, once available, is something to consume. We show this fails: naively appending a caption can lower accuracy rather than raise it, dropping Qwen2.5-VL-3B† on HallusionBench by nearly ten points. To understand why, we build GD-Probe, a diagnostic set that pairs a global and a detail question on the same image, so that any difference in caption effect is attributable to the question alone. Caption utility proves to be a per-query property: the same caption helps global questions and harms detail ones, through a single mechanism -- an embedded caption competes with the image for attention and pulls the model's evidence onto its own text -- whose sign is set by whether the caption covers the queried content. Crucially, this regime is readable from quantities the decoder already emits, with no attention access or grounding. We turn this into GEASS (Gated Evidence-Adaptive Selective Caption Trust), a training-free, logit-level module that decides per query how much of the caption to trust, gating it by the clean path's confidence, weighting it by the entropy reduction it induces, and raising the evidence bar when the two pathways disagree. Across four VLMs and two benchmarks (POPE and HallusionBench), GEASS improves over both vanilla inference and contrastive decoding under a single fixed setting, adding only two forward passes and no parameters.

翻译：视觉-语言模型（VLM）会幻觉出实际不存在的物体，为遏制这一问题，越来越多的研究尝试将模型自身生成的描述作为辅助证据输入——其隐含假设是：一旦生成描述，该描述便成为可被利用的信息。但我们的研究表明这种策略存在缺陷：简单附加描述非但未能提升准确性，反而会降低模型性能，例如在HallusionBench基准测试中，Qwen2.5-VL-3B†的性能下降近十个百分点。为理解根本原因，我们构建了诊断数据集GD-Probe，该数据集为每张图像配对全局性问题和细节性问题，使得描述效果的差异可完全归因于问题类型。实验证明，描述的有效性呈现逐查询依赖特性：同一段描述通过单一机制——嵌入的描述会与图像竞争注意力资源，将模型证据重心拉向其自身文本——同时提升全局性问题表现但损害细节性问题表现，其作用方向取决于描述是否覆盖查询内容。关键在于，这一作用模式可通过解码器已输出的特征量直接判读，无需借助注意力机制或基础事实。基于此发现，我们提出GEASS（门控证据自适应选择性描述信任机制），这是一种无需训练、在logit层面运行的模块，它通过以下方式按查询级别决策描述的可信度：以原始推理路径的置信度作为门控信号，根据描述引起的熵减程度进行加权，当两条推理路径产生分歧时提高证据门槛。在四个视觉-语言模型和两个基准测试（POPE与HallusionBench）上，GEASS在单一固定配置下同时超越标准推理和对比解码方法，仅需额外两次前向传播且无需引入任何参数。