Recent advancements in large vision-language models (LVLMs) have demonstrated impressive capability in visual information understanding with human language. Despite these advances, LVLMs still face challenges with multimodal hallucination, such as generating text descriptions of objects that are not present in the visual information. However, the underlying fundamental reasons of multimodal hallucinations remain poorly explored. In this paper, we propose a new perspective, suggesting that the inherent biases in LVLMs might be a key factor in hallucinations. Specifically, we systematically identify a semantic shift bias related to paragraph breaks ('$\textbackslash n\textbackslash n$'), where the content before and after '$\textbackslash n\textbackslash n$' in the training data frequently exhibit significant semantic changes. This pattern leads the model to infer that the contents following '$\textbackslash n\textbackslash n$' should be obviously different from the preceding contents with less hallucinatory descriptions, thereby increasing the probability of hallucinatory descriptions subsequent to the '$\textbackslash n\textbackslash n$'. We have validated this hypothesis on multiple publicly available LVLMs. Besides, we find that deliberately inserting '$\textbackslash n\textbackslash n$' at the generated description can induce more hallucinations. A simple method is proposed to effectively mitigate the hallucination of LVLMs by skipping the output of `\textbackslash n'.

翻译：近期大型视觉语言模型（LVLMs）的进展展示了其结合人类语言理解视觉信息的卓越能力。尽管取得这些进步，LVLMs仍面临多模态幻觉的挑战，例如生成视觉信息中不存在对象的文本描述。然而，多模态幻觉的根本原因仍缺乏深入探索。本文提出新观点，认为LVLMs的内在偏差可能是导致幻觉的关键因素。具体而言，我们系统性地识别出与段落分隔符（'$\backslash n\backslash n$'）相关的语义偏移偏差——训练数据中'$\backslash n\backslash n$'前后的内容常呈现显著语义变化。这种模式引导模型推断'$\backslash n\backslash n$'之后的内容应与前文存在明显差异且更易产生幻觉性描述，从而增加'$\backslash n\backslash n$'后出现幻觉描述的概率。我们在多个公开LVLMs上验证了这一假设。此外，我们发现刻意在生成描述中插入'$\backslash n\backslash n$'会诱发更多幻觉。我们提出通过跳过`\backslash n'输出来有效缓解LVLMs幻觉的简单方法。