Multimodal large language models (MLLMs) contribute a powerful mechanism to understanding visual information building on large language models. However, MLLMs are notorious for suffering from hallucinations, especially when generating lengthy, detailed descriptions for images. Our analysis reveals that hallucinations stem from the inherent summarization mechanism of large language models, leading to excessive dependence on linguistic tokens while neglecting vision information. In this paper, we propose NoiseBoost, a broadly applicable and simple method for alleviating hallucinations for MLLMs through the integration of noise feature perturbations. Noise perturbation acts as a regularizer, facilitating a balanced distribution of attention weights among visual and linguistic tokens. Despite its simplicity, NoiseBoost consistently enhances the performance of MLLMs across common training strategies, including supervised fine-tuning and reinforcement learning. Further, NoiseBoost pioneerly enables semi-supervised learning for MLLMs, unleashing the power of unlabeled data. Comprehensive experiments demonstrate that NoiseBoost improves dense caption accuracy by 8.1% with human evaluation and achieves comparable results with 50% of the data by mining unlabeled data. Code and models are available at https://kaiwu5.github.io/noiseboost.

翻译：多模态大语言模型（MLLMs）在大型语言模型的基础上，为理解视觉信息提供了一种强大的机制。然而，MLLMs因产生幻觉而备受诟病，尤其是在为图像生成长篇详细描述时。我们的分析表明，幻觉源于大型语言模型固有的摘要生成机制，导致模型过度依赖语言标记而忽视视觉信息。本文提出NoiseBoost，一种广泛适用且简单的方法，通过引入噪声特征扰动来缓解MLLMs的幻觉问题。噪声扰动作为一种正则化器，有助于在视觉标记和语言标记之间实现注意力权重的均衡分布。尽管方法简单，NoiseBoost在包括监督微调和强化学习在内的常见训练策略中，均能持续提升MLLMs的性能。此外，NoiseBoost首次实现了MLLMs的半监督学习，释放了未标注数据的潜力。综合实验表明，NoiseBoost通过人工评估将密集描述准确率提升了8.1%，并通过挖掘未标注数据，仅用50%的数据量即达到了可比的结果。代码和模型可在 https://kaiwu5.github.io/noiseboost 获取。