While Vision-Language Models (VLMs) have garnered increasing attention in the AI community due to their promising practical applications, they exhibit persistent hallucination issues, generating outputs misaligned with visual inputs. Recent studies attribute these hallucinations to VLMs' over-reliance on linguistic priors and insufficient visual feature integration, proposing heuristic decoding calibration strategies to mitigate them. However, the non-trainable nature of these strategies inherently limits their optimization potential. To this end, we propose an adversarial parametric editing framework for Hallucination mitigation in VLMs, which follows an \textbf{A}ctivate-\textbf{L}ocate-\textbf{E}dit \textbf{A}dversarially paradigm. Specifically, we first construct an activation dataset that comprises grounded responses (positive samples attentively anchored in visual features) and hallucinatory responses (negative samples reflecting LLM prior bias and internal knowledge artifacts). Next, we identify critical hallucination-prone parameter clusters by analyzing differential hidden states of response pairs. Then, these clusters are fine-tuned using prompts injected with adversarial tuned prefixes that are optimized to maximize visual neglect, thereby forcing the model to prioritize visual evidence over inherent parametric biases. Evaluations on both generative and discriminative VLM tasks demonstrate the significant effectiveness of ALEAHallu in alleviating hallucinations. Our code is available at https://github.com/hujiayu1223/ALEAHallu.

翻译：尽管视觉语言模型（VLMs）因其广阔的实际应用前景在人工智能领域受到越来越多的关注，但它们表现出持续的幻觉问题，即生成与视觉输入不一致的输出。最近的研究将这些幻觉归因于VLMs对语言先验的过度依赖以及视觉特征整合不足，并提出了启发式解码校准策略来缓解这些问题。然而，这些策略的不可训练性本质上限制了其优化潜力。为此，我们提出了一种用于缓解VLM幻觉的对抗性参数编辑框架，该框架遵循一种**激**活-**定**位-**对**抗**编**辑的范式。具体而言，我们首先构建一个激活数据集，其中包含基于视觉的响应（积极样本，其注意力锚定在视觉特征上）和幻觉响应（消极样本，反映LLM先验偏差和内部知识伪影）。接着，我们通过分析响应对的差分隐藏状态，识别出关键的易幻觉参数簇。然后，使用注入了对抗性调优前缀的提示对这些参数簇进行微调，这些前缀经过优化以最大化视觉忽视，从而迫使模型优先考虑视觉证据而非固有的参数偏差。在生成式和判别式VLM任务上的评估表明，ALEAHallu在缓解幻觉方面具有显著的有效性。我们的代码可在 https://github.com/hujiayu1223/ALEAHallu 获取。