We present DRESS, a large vision language model (LVLM) that innovatively exploits Natural Language feedback (NLF) from Large Language Models to enhance its alignment and interactions by addressing two key limitations in the state-of-the-art LVLMs. First, prior LVLMs generally rely only on the instruction finetuning stage to enhance alignment with human preferences. Without incorporating extra feedback, they are still prone to generate unhelpful, hallucinated, or harmful responses. Second, while the visual instruction tuning data is generally structured in a multi-turn dialogue format, the connections and dependencies among consecutive conversational turns are weak. This reduces the capacity for effective multi-turn interactions. To tackle these, we propose a novel categorization of the NLF into two key types: critique and refinement. The critique NLF identifies the strengths and weaknesses of the responses and is used to align the LVLMs with human preferences. The refinement NLF offers concrete suggestions for improvement and is adopted to improve the interaction ability of the LVLMs-- which focuses on LVLMs' ability to refine responses by incorporating feedback in multi-turn interactions. To address the non-differentiable nature of NLF, we generalize conditional reinforcement learning for training. Our experimental results demonstrate that DRESS can generate more helpful (9.76%), honest (11.52%), and harmless (21.03%) responses, and more effectively learn from feedback during multi-turn interactions compared to SOTA LVMLs.

翻译：我们提出DRESS，一种大型视觉语言模型（LVLM），其创新地利用大型语言模型的自然语言反馈（NLF）来增强对齐与交互能力，解决了当前最先进LVLM的两个关键局限性。首先，以往的LVLM通常仅依赖指令微调阶段来增强与人类偏好的对齐。由于未纳入额外反馈，它们仍容易生成无帮助、幻觉或有害的响应。其次，尽管视觉指令微调数据通常以多轮对话格式组织，但连续对话轮次之间的连接和依赖性较弱，这降低了有效多轮交互的能力。为解决这些问题，我们提出将NLF创新性地分为两种关键类型：批评和优化。批评型NLF识别响应的优缺点，用于使LVLM与人类偏好对齐；优化型NLF提供具体的改进建议，用于提升LVLM的交互能力——其重点在于通过多轮交互中融入反馈来优化响应。针对NLF不可微的特性，我们泛化条件强化学习进行训练。实验结果表明，与最先进的LVLM相比，DRESS能生成更有帮助（提升9.76%）、更诚实（提升11.52%）和更无害（提升21.03%）的响应，并在多轮交互中更有效地从反馈中学习。