In the realms of computer vision and natural language processing, Large Vision-Language Models (LVLMs) have become indispensable tools, proficient in generating textual descriptions based on visual inputs. Despite their advancements, our investigation reveals a noteworthy bias in the generated content, where the output is primarily influenced by the underlying Large Language Models (LLMs) prior rather than the input image. Our empirical experiments underscore the persistence of this bias, as LVLMs often provide confident answers even in the absence of relevant images or given incongruent visual input. To rectify these biases and redirect the model's focus toward vision information, we introduce two simple, training-free strategies. Firstly, for tasks such as classification or multi-choice question-answering (QA), we propose a ``calibration'' step through affine transformation to adjust the output distribution. This ``Post-Hoc debias'' approach ensures uniform scores for each answer when the image is absent, serving as an effective regularization technique to alleviate the influence of LLM priors. For more intricate open-ended generation tasks, we extend this method to ``Debias sampling'', drawing inspirations from contrastive decoding methods. Furthermore, our investigation sheds light on the instability of LVLMs across various decoding configurations. Through systematic exploration of different settings, we significantly enhance performance, surpassing reported results and raising concerns about the fairness of existing evaluations. Comprehensive experiments substantiate the effectiveness of our proposed strategies in mitigating biases. These strategies not only prove beneficial in minimizing hallucinations but also contribute to the generation of more helpful and precise illustrations.

翻译：在计算机视觉和自然语言处理领域，大型视觉语言模型（LVLMs）已成为不可或缺的工具，能够根据视觉输入生成文本描述。尽管取得了进展，我们的研究发现生成内容中存在显著的偏见，即输出主要受底层大语言模型（LLMs）先验知识的影响，而非输入图像。实证实验表明，这种偏见持续存在，例如LVLMs在缺少相关图像或面对不一致的视觉输入时，仍会给出自信的答案。为纠正这些偏见并将模型关注点重新导向视觉信息，我们提出了两种无需训练的简单策略。首先，针对分类或多选问答（QA）等任务，我们提出通过仿射变换进行“校准”步骤以调整输出分布。这种“事后去偏”方法确保在缺少图像时每个答案获得均匀分数，作为一种有效的正则化技术，减轻了LLM先验知识的影响。对于更复杂的开放式生成任务，我们将该方法扩展为“去偏采样”，借鉴对比解码方法的思路。此外，我们的研究揭示了LVLMs在不同解码配置下的不稳定性。通过系统探索不同设置，我们显著提升了性能，超越了已有报道的结果，并对现有评估的公平性提出质疑。大量实验证实了我们提出的去偏策略的有效性。这些策略不仅有助于减少幻觉，还能生成更有益且更精确的说明。