Large Vision-Language Models (LVLMs) have demonstrated proficiency in tackling a variety of visual-language tasks. However, current LVLMs suffer from misalignment between text and image modalities which causes three kinds of hallucination problems, i.e., object existence, object attribute, and object relationship. To tackle this issue, existing methods mainly utilize Reinforcement Learning (RL) to align modalities in LVLMs. However, they still suffer from three main limitations: (1) General feedback can not indicate the hallucination type contained in the response; (2) Sparse rewards only give the sequence-level reward for the whole response; and (3)Annotation cost is time-consuming and labor-intensive. To handle these limitations, we propose an innovative method to align modalities in LVLMs through Fine-Grained Artificial Intelligence Feedback (FGAIF), which mainly consists of three steps: AI-based Feedback Collection, Fine-grained Reward Model Training, and Reinforcement Learning with Fine-grained Reward. Specifically, We first utilize AI tools to predict the types of hallucination for each segment in the response and obtain a collection of fine-grained feedback. Then, based on the collected reward data, three specialized reward models are trained to produce dense rewards. Finally, a novel fine-grained feedback module is integrated into the Proximal Policy Optimization (PPO) algorithm. Extensive experiments are conducted on hallucination and general benchmarks, demonstrating the superior performance of our proposed method. Notably, compared with previous models trained with the RL-based aligning method, our proposed method is effective even with fewer parameters.

翻译：大型视觉-语言模型（LVLMs）在处理多种视觉-语言任务中展现出卓越能力。然而，当前LVLMs存在文本与图像模态之间的错位问题，这导致三类幻觉现象，即物体存在性、物体属性及物体关系。为解决此问题，现有方法主要采用强化学习（RL）对齐LVLMs中的模态。然而，这些方法仍存在三大局限性：（1）通用反馈无法指示响应中包含的幻觉类型；（2）稀疏奖励仅对整体响应提供序列级奖励；（3）标注成本耗时且劳动密集。为应对这些局限，我们提出一种创新方法——通过细粒度人工智能反馈（FGAIF）对齐LVLMs中的模态，该方法主要包括三个步骤：基于AI的反馈收集、细粒度奖励模型训练及基于细粒度奖励的强化学习。具体而言，我们首先利用AI工具预测响应中每个片段对应的幻觉类型，并获取细粒度反馈集合。然后，基于收集的奖励数据，训练三个专用奖励模型以生成密集奖励。最后，将新颖的细粒度反馈模块集成到近端策略优化（PPO）算法中。在幻觉检测与通用基准上进行了广泛实验，结果表明我们提出的方法具有优越性能。值得注意的是，与以往基于RL对齐方法训练的模型相比，本方法即使在参数更少的情况下依然有效。