Despite the significant progress of Multimodal Large Language Models (MLLMs) across diverse tasks, hallucination -- corresponding to the generation of visually inconsistent objects, attributes, or relations -- remains a major obstacle to their reliable deployment. Unlike pure language models, MLLMs must ground their generation process in visual inputs. However, existing models often suffer from semantic drift during decoding, causing outputs to diverge from visual facts as the sequence length increases. To address this issue, we propose KVSmooth, a training-free and plug-and-play method that mitigates hallucination by performing attention-entropy-guided adaptive smoothing on hidden states. Specifically, KVSmooth applies an exponential moving average (EMA) to both keys and values in the KV-Cache, while dynamically quantifying the sink degree of each token through the entropy of its attention distribution to adaptively adjust the smoothing strength. Unlike computationally expensive retraining or contrastive decoding methods, KVSmooth operates efficiently during inference without additional training or model modification. Extensive experiments demonstrate that KVSmooth significantly reduces hallucination ($\mathit{CHAIR}_{S}$ from $41.8 \rightarrow 18.2$) while improving overall performance ($F_1$ score from $77.5 \rightarrow 79.2$), achieving higher precision and recall simultaneously. In contrast, prior methods often improve one at the expense of the other, validating the effectiveness and generality of our approach.

翻译：尽管多模态大语言模型（MLLMs）在多种任务上取得了显著进展，但幻觉问题——即生成与视觉内容不一致的物体、属性或关系——仍然是其可靠部署的主要障碍。与纯语言模型不同，MLLMs必须将其生成过程基于视觉输入。然而，现有模型在解码过程中常遭受语义漂移，导致输出随着序列长度增加而偏离视觉事实。为解决这一问题，我们提出KVSmooth，一种无需训练且即插即用的方法，通过对隐藏状态进行注意力熵引导的自适应平滑来缓解幻觉。具体而言，KVSmooth对KV-Cache中的键和值应用指数移动平均（EMA），同时通过每个词元注意力分布的熵动态量化其“沉没”程度，以自适应调整平滑强度。与计算成本高昂的重新训练或对比解码方法不同，KVSmooth在推理过程中高效运行，无需额外训练或模型修改。大量实验表明，KVSmooth显著减少了幻觉（$\mathit{CHAIR}_{S}$从$41.8 \rightarrow 18.2$），同时提升了整体性能（$F_1$分数从$77.5 \rightarrow 79.2$），实现了更高的精确率和召回率。相比之下，先前的方法往往以牺牲其中一项指标为代价来提升另一项，这验证了我们方法的有效性和普适性。