Large vision-language models can produce object hallucinations in image descriptions, highlighting the need for effective detection and mitigation strategies. Prior work commonly relies on the model's attention weights on visual tokens as a detection signal. We reveal that coarse-grained attention-based analysis is unreliable due to hidden confounders, specifically token position and object repetition in a description. This leads to Simpson's paradox: the attention trends reverse or disappear when statistics are aggregated. Based on this observation, we introduce HaloProbe, a Bayesian framework that factorizes external description statistics and internal decoding signals to estimate token-level hallucination probabilities. HaloProbe uses balanced training to isolate internal evidence and combines it with learned prior over external features to recover the true posterior. While intervention-based mitigation methods often degrade utility or fluency by modifying models' internals, we use HaloProbe as an external scoring signal for non-invasive mitigation. Our experiments show that HaloProbe-guided decoding reduces hallucinations more effectively than state-of-the-art intervention-based methods while preserving utility.

翻译：大型视觉语言模型在图像描述中可能产生物体幻觉，这凸显了有效检测与缓解策略的必要性。现有方法通常依赖模型对视觉令牌的注意力权重作为检测信号。我们揭示出基于粗粒度注意力的方法因隐藏混杂因素（尤其是描述中令牌位置与物体重复）而不可靠，导致辛普森悖论：当统计量聚合时，注意力趋势出现反转或消失。基于此发现，我们提出HaloProbe——一种贝叶斯框架，通过分解外部描述统计量与内部解码信号来估计令牌级幻觉概率。该方法采用平衡训练隔离内部证据，并将其与外部特征上的先验学习相结合以恢复真实后验。虽然基于干预的缓解方法常通过修改模型内部机制而损害实用性或流畅性，但我们利用HaloProbe作为外部评分信号实现非侵入式缓解。实验表明，HaloProbe引导解码在保持实用性的同时，比最先进的基于干预的方法更有效地减少幻觉。