We present DRESS, a large vision language model (LVLM) that innovatively exploits Natural Language feedback (NLF) from Large Language Models to enhance its alignment and interactions by addressing two key limitations in the state-of-the-art LVLMs. First, prior LVLMs generally rely only on the instruction finetuning stage to enhance alignment with human preferences. Without incorporating extra feedback, they are still prone to generate unhelpful, hallucinated, or harmful responses. Second, while the visual instruction tuning data is generally structured in a multi-turn dialogue format, the connections and dependencies among consecutive conversational turns are weak. This reduces the capacity for effective multi-turn interactions. To tackle these, we propose a novel categorization of the NLF into two key types: critique and refinement. The critique NLF identifies the strengths and weaknesses of the responses and is used to align the LVLMs with human preferences. The refinement NLF offers concrete suggestions for improvement and is adopted to improve the interaction ability of the LVLMs-- which focuses on LVLMs' ability to refine responses by incorporating feedback in multi-turn interactions. To address the non-differentiable nature of NLF, we generalize conditional reinforcement learning for training. Our experimental results demonstrate that DRESS can generate more helpful (9.76%), honest (11.52%), and harmless (21.03%) responses, and more effectively learn from feedback during multi-turn interactions compared to SOTA LVMLs.

翻译：我们提出DRESS，这是一种大型视觉-语言模型（LVLM），它创新性地利用来自大型语言模型的自然语言反馈（NLF），通过解决当前最先进LVLM的两个关键局限性来增强其对齐和交互能力。第一，先前的LVLM通常仅依赖指令微调阶段来增强与人类偏好的对齐。由于没有融入额外反馈，它们仍容易生成无帮助、幻觉或有害的回应。第二，尽管视觉指令微调数据通常以多轮对话格式组织，但连续对话轮次之间的连接和依赖性较弱，这削弱了有效多轮交互的能力。为解决这些问题，我们提出将NLF创新性地分为两种关键类型：批评与改进。批评型NLF识别回应的优缺点，用于使LVLM与人类偏好对齐；改进型NLF提供具体的改进建议，用于提升LVLM的交互能力——即LVLM在多轮交互中通过融入反馈来优化回应的能力。为应对NLF不可微分的特性，我们将条件强化学习泛化用于训练。实验结果表明，与最先进的LVLM相比，DRESS能生成更有帮助（9.76%）、更诚实（11.52%）且更无害（21.03%）的回应，并在多轮交互中更有效地从反馈中学习。