Hallucination, posed as a pervasive challenge of multi-modal large language models (MLLMs), has significantly impeded their real-world usage that demands precise judgment. Existing methods mitigate this issue with either training with specific designed data or inferencing with external knowledge from other sources, incurring inevitable additional costs. In this paper, we present OPERA, a novel MLLM decoding method grounded in an Over-trust Penalty and a Retrospection-Allocation strategy, serving as a nearly free lunch to alleviate the hallucination issue without additional data, knowledge, or training. Our approach begins with an interesting observation that, most hallucinations are closely tied to the knowledge aggregation patterns manifested in the self-attention matrix, i.e., MLLMs tend to generate new tokens by focusing on a few summary tokens, but not all the previous tokens. Such partial over-trust inclination results in the neglecting of image tokens and describes the image content with hallucination. Based on the observation, OPERA introduces a penalty term on the model logits during the beam-search decoding to mitigate the over-trust issue, along with a rollback strategy that retrospects the presence of summary tokens in the previously generated tokens, and re-allocate the token selection if necessary. With extensive experiments, OPERA shows significant hallucination-mitigating performance on different MLLMs and metrics, proving its effectiveness and generality. Our code is available at: https://github.com/shikiw/OPERA.

翻译：摘要：幻觉作为多模态大语言模型（MLLMs）中普遍存在的挑战，严重阻碍了其对精确判断要求严苛的实际应用。现有方法通过使用特定设计数据进行训练或借助其他来源的外部知识进行推理来缓解该问题，但不可避免地产生了额外成本。本文提出OPERA——一种基于过度信任惩罚与回溯分配策略的新型MLLM解码方法，作为近乎"免费午餐"的解决方案，无需额外数据、知识或训练即可缓解幻觉问题。我们的方法始于一个有趣发现：多数幻觉与自注意力矩阵中呈现的知识聚合模式密切相关，即MLLMs倾向于通过聚焦少量摘要标记而非全部先前标记来生成新标记。这种局部过度信任倾向导致图像标记被忽略，从而在描述图像内容时产生幻觉。基于该发现，OPERA在束搜索解码过程中对模型对数概率施加惩罚项以缓解过度信任问题，同时引入回溯机制——检测先前生成标记中摘要标记的存在，必要时重新分配标记选择。经广泛实验验证，OPERA在不同MLLMs和评估指标上均展现出显著的幻觉缓解性能，验证了其有效性与通用性。我们的代码见：https://github.com/shikiw/OPERA。