Vision-language models (VLMs) raise growing concerns about privacy, copyright, and bias, motivating machine unlearning to remove sensitive knowledge. However, existing methods primarily fine-tune the language decoder, leading to superficial forgetting that fails to erase underlying visual representations and often introduces object hallucination. We propose HFRU, a reinforcement unlearning framework that operates on the vision encoder for deep semantic removal. Our two-stage approach combines alignment disruption with GRPO-based optimization using a composite reward, including an abstraction reward that encourages semantically valid substitutions and mitigates hallucinations. Experiments on object recognition and face identity tasks show that HFRU achieves over 98% forgetting and retention performance, while introducing negligible object hallucination, significantly outperforming prior methods.Our code and implementation details are available at https://github.com/XMUDeepLIT/HFRU.

翻译：[翻译摘要] 视觉-语言模型（VLM）引发了关于隐私、版权和偏见日益增长的担忧，促使研究者通过机器遗忘技术移除敏感知识。然而，现有方法主要对语言解码器进行微调，导致遗忘效果浅层化，既无法擦除底层视觉表征，又常引发物体幻觉问题。本文提出HFRU——一种基于强化学习的遗忘框架，通过作用于视觉编码器实现深度语义移除。该两阶段方法结合了对齐破坏机制与基于GRPO的优化，并采用复合奖励函数，其中抽象奖励鼓励语义有效替换并抑制幻觉。在物体识别和人脸身份任务上的实验表明，HFRU实现了超98%的遗忘与保持性能，同时引入可忽略的物体幻觉，显著优于此前方法。本文代码与实现细节详见https://github.com/XMUDeepLIT/HFRU。