Hallucinations in Large Vision-Language Models (LVLMs) remain a persistent challenge, often stemming from inadequate integration of visual information during multimodal reasoning. A key cause is the model's over-reliance on textual priors and underutilization of visual cues, leading to outputs that are linguistically fluent but visually inaccurate. For example, given an image of an empty kitchen countertop, an LVLM might hallucinate a "bowl of fruit" or "cup of coffee", relying on language associations rather than visual evidence. Most LVLMs incorporate visual features by appending them to the input stream of a pre-trained LLM and training on large-scale vision-language datasets. Our systematic analysis reveals that this strategy often leads to over-dependence on textual information due to the inherent bias of LLMs towards language-dominant representations. This imbalance skews attention towards the text over visual content, weakening the model's ability to ground outputs in visual inputs. To address this, we propose a simple yet effective visual feature incorporation method that encourages the model to learn visually-informed textual embeddings distinct from those of the base LLM and promotes a more balanced attention distribution. Experimental results across multiple hallucination benchmarks demonstrate that our method significantly reduces hallucinations and fosters more balanced multimodal reasoning. Notably, our approach achieves substantial gains, including +9.33% on MMVP-MLLM, +2.99% on POPE-AOKVQA, up to +3.4% on Merlin, and +3% on the hard-data split of HallusionBench.

翻译：大型视觉-语言模型（LVLMs）中的幻觉现象仍是一个持续存在的挑战，其根源常在于多模态推理过程中视觉信息整合不足。关键原因在于模型过度依赖文本先验知识而未能充分利用视觉线索，导致输出虽语言流畅却与视觉内容不符。例如，面对一张空厨房台面的图像，LVLM可能基于语言关联而非视觉证据，幻觉出“一碗水果”或“一杯咖啡”。多数LVLM通过将视觉特征附加到预训练大语言模型（LLM）的输入流中，并在大规模视觉-语言数据集上训练来实现视觉特征的融合。我们的系统分析表明，由于LLM对语言主导表征存在固有偏见，这种策略常导致模型过度依赖文本信息。这种不平衡会促使注意力偏向文本而非视觉内容，削弱模型基于视觉输入生成输出的能力。为解决该问题，我们提出一种简单而有效的视觉特征融合方法，促使模型学习区别于基础LLM的视觉感知型文本嵌入，并推动更均衡的注意力分布。在多个幻觉评测基准上的实验结果表明，我们的方法显著减少幻觉现象，促进更平衡的多模态推理。值得注意的是，本方法在MMVP-MLLM、POPE-AOKVQA、Merlin和HallusionBench硬数据子集上分别取得+9.33%、+2.99%、最高+3.4%和+3%的性能提升。