Large vision-language models (VLMs) are highly capable, yet often hallucinate by favoring textual prompts over visual evidence. We study this failure mode in a controlled object-counting setting, where the prompt overstates the number of objects in the image (e.g., asking a model to describe four waterlilies when only three are present). At low object counts, models often correct the overestimation, but as the number of objects increases, they increasingly conform to the prompt regardless of the discrepancy. Through mechanistic analysis of three VLMs, we identify a small set of attention heads whose ablation substantially reduces prompt-induced hallucinations (PIH) by at least 40% without additional training. Across models, PIH-heads mediate prompt copying in model-specific ways. We characterize these differences and show that PIH ablation increases correction toward visual evidence. Our findings offer insights into the internal mechanisms driving prompt-induced hallucinations, revealing model-specific differences in how these behaviors are implemented.

翻译：大型视觉语言模型（VLMs）虽然能力强大，但常常出现幻觉，倾向于依赖文本提示而非视觉证据。我们在一个受控的对象计数场景中研究这种失效模式，其中提示会夸大图像中对象的数量（例如，当图像中仅存在三朵睡莲时，要求模型描述四朵睡莲）。在对象数量较少时，模型通常会纠正这种高估，但随着对象数量的增加，无论差异如何，模型都越来越倾向于遵从提示。通过对三种VLMs进行机制分析，我们识别出一小组注意力头，对其执行消融操作可在无需额外训练的情况下，将提示诱导幻觉（PIH）显著降低至少40%。在不同模型中，PIH头以模型特定的方式介导了提示复制。我们描述了这些差异，并表明PIH消融能增强模型向视觉证据的校正。我们的研究结果为了解驱动提示诱导幻觉的内部机制提供了见解，揭示了这些行为在实现方式上的模型特异性差异。