Recent studies have shown that large vision-language models (LVLMs) often suffer from the issue of object hallucinations (OH). To mitigate this issue, we introduce an efficient method that edits the model weights based on an unsafe subspace, which we call HalluSpace in this paper. With truthful and hallucinated text prompts accompanying the visual content as inputs, the HalluSpace can be identified by extracting the hallucinated embedding features and removing the truthful representations in LVLMs. By orthogonalizing the model weights, input features will be projected into the Null space of the HalluSpace to reduce OH, based on which we name our method Nullu. We reveal that HalluSpaces generally contain statistical bias and unimodal priors of the large language models (LLMs) applied to build LVLMs, which have been shown as essential causes of OH in previous studies. Therefore, null space projection suppresses the LLMs' priors to filter out the hallucinated features, resulting in contextually accurate outputs. Experiments show that our method can effectively mitigate OH across different LVLM families without extra inference costs and also show strong performance in general LVLM benchmarks. Code is released at \url{https://github.com/Ziwei-Zheng/Nullu}.

翻译：近期研究表明，大型视觉语言模型（LVLM）常受物体幻觉（OH）问题困扰。为缓解此问题，我们提出一种基于不安全子空间编辑模型权重的有效方法，本文将该子空间称为HalluSpace。通过将伴随视觉内容的真实文本提示与幻觉文本提示作为输入，可从LVLM中提取幻觉嵌入特征并移除真实表征，从而识别HalluSpace。通过对模型权重进行正交化处理，输入特征将被投影至HalluSpace的零空间以减少OH，据此我们将本方法命名为Nullu。我们发现HalluSpace普遍包含用于构建LVLM的大型语言模型（LLM）的统计偏差与单模态先验，而先前研究已证明这些是导致OH的关键成因。因此，零空间投影通过抑制LLM先验来过滤幻觉特征，从而生成上下文准确的输出。实验表明，本方法能有效缓解不同LVLM系列中的OH问题，且无需额外推理成本，同时在通用LVLM基准测试中表现出强劲性能。代码发布于\url{https://github.com/Ziwei-Zheng/Nullu}。