In the realms of computer vision and natural language processing, Large Vision-Language Models (LVLMs) have become indispensable tools, proficient in generating textual descriptions based on visual inputs. Despite their advancements, our investigation reveals a noteworthy bias in the generated content, where the output is primarily influenced by the underlying Large Language Models (LLMs) prior rather than the input image. Our empirical experiments underscore the persistence of this bias, as LVLMs often provide confident answers even in the absence of relevant images or given incongruent visual input. To rectify these biases and redirect the model's focus toward vision information, we introduce two simple, training-free strategies. Firstly, for tasks such as classification or multi-choice question-answering (QA), we propose a ``calibration'' step through affine transformation to adjust the output distribution. This ``Post-Hoc debias'' approach ensures uniform scores for each answer when the image is absent, serving as an effective regularization technique to alleviate the influence of LLM priors. For more intricate open-ended generation tasks, we extend this method to ``Debias sampling'', drawing inspirations from contrastive decoding methods. Furthermore, our investigation sheds light on the instability of LVLMs across various decoding configurations. Through systematic exploration of different settings, we significantly enhance performance, surpassing reported results and raising concerns about the fairness of existing evaluations. Comprehensive experiments substantiate the effectiveness of our proposed strategies in mitigating biases. These strategies not only prove beneficial in minimizing hallucinations but also contribute to the generation of more helpful and precise illustrations.

翻译：在计算机视觉和自然语言处理领域，大型视觉语言模型已成为不可或缺的工具，能够基于视觉输入生成文本描述。尽管它们取得了进展，但我们的研究发现生成内容中存在显著的偏差，即输出主要受底层大型语言模型的先验知识影响，而非输入图像。实验证实了这种偏差的持续性：即便在缺乏相关图像或输入不匹配的视觉信息时，LVLMs仍会给出自信的回答。为纠正这些偏差并将模型注意力重新聚焦于视觉信息，我们提出了两种无需训练的简易策略。首先，针对分类或多选题等任务，我们提出通过仿射变换调整输出分布的“校准”步骤。这种“事后去偏”方法可在图像缺失时确保每个答案获得均等分数，成为缓解LLM先验影响的有效正则化技术。对于更复杂的开放式生成任务，我们借鉴对比解码方法将其扩展为“去偏采样”。此外，研究发现LVLMs在不同解码配置下存在不稳定性。通过系统探索不同设置，我们显著提升性能并超越已报告成果，引发对现有评估公平性的思考。全面实验验证了所提策略在缓解偏差方面的有效性——这些策略不仅有助于减少幻觉现象，还能生成更实用且精确的图文描述。