Despite their impressive performance across a wide range of tasks, Large Vision-Language Models (LVLMs) remain prone to hallucination. In this study, we propose a comprehensive intervention framework aligned with the transformer's causal architecture in LVLMs, integrating the effects of different intervention paths on hallucination. We find that hallucinations in LVLMs do not arise from a single causal path, but rather from the interplay among image-to-input-text, image-to-output-text, and text-to-text pathways. For the first time, we also find that LVLMs rely on different pathways depending on the question-answer alignment format. Building on these insights, we propose simple yet effective methods to identify and intervene on critical hallucination heads within each pathway, tailored to discriminative and generative formats. Experiments across multiple benchmarks demonstrate that our approach consistently reduces hallucinations across diverse alignment types.

翻译：尽管大型视觉语言模型（LVLM）在广泛任务中展现出卓越性能，但其仍易产生幻觉现象。本研究提出一个与LVLM中Transformer因果架构相协调的综合干预框架，整合了不同干预路径对幻觉的影响。我们发现，LVLM中的幻觉并非源于单一因果路径，而是由图像到输入文本、图像到输出文本以及文本到文本三条路径间的相互作用所导致。我们首次发现，LVLM会根据问答对齐格式的不同而依赖不同的路径。基于这些发现，我们针对判别式与生成式格式，提出了简单而有效的方法来识别并干预各路径中的关键幻觉头。在多个基准测试上的实验表明，我们的方法能持续降低不同对齐类型下的幻觉现象。