The diversity of contexts in which large language models (LLMs) are deployed requires the ability to modify or customize default model behaviors to incorporate nuanced requirements and preferences. A convenient interface to specify such model adjustments is high-level verbal feedback, such as "Don't use emojis when drafting emails to my boss." However, while writing high-level feedback is far simpler than collecting annotations for reinforcement learning from human feedback (RLHF), we find that simply prompting a model with such feedback leads to overgeneralization of the feedback to contexts where it is not relevant. We study the problem of incorporating verbal feedback without such overgeneralization, inspiring a new method Contextualized Critiques with Constrained Preference Optimization (C3PO). C3PO uses a piece of high-level feedback to generate a small synthetic preference dataset specifying how the feedback should (and should not) be applied. It then fine-tunes the model in accordance with the synthetic preference data while minimizing the divergence from the original model for prompts where the feedback does not apply. Our experimental results indicate that our approach effectively applies verbal feedback to relevant scenarios while preserving existing behaviors for other contexts. For both human- and GPT-4-generated high-level feedback, C3PO effectively adheres to the given feedback comparably to in-context baselines while reducing overgeneralization by 30%.

翻译：大型语言模型（LLMs）部署场景的多样性要求能够修改或自定义默认模型行为，以融入细致的需求与偏好。指定此类模型调整的便捷接口是高级言语反馈，例如“给老板写邮件时不要使用表情符号”。然而，尽管编写高级反馈远比收集用于人类反馈强化学习（RLHF）的标注简单，我们发现直接使用此类反馈提示模型会导致反馈过度泛化到不相关场景中。我们研究了在不引发过度泛化的情况下融入言语反馈的问题，并提出了一种新方法——基于约束偏好优化的情境化批评（C3PO）。C3PO利用一条高级反馈生成一个合成偏好数据集，明确指定反馈应（及不应）应用的方式。随后，它根据该合成偏好数据对模型进行微调，同时最小化模型在不适用反馈的提示上与原始模型的偏差。实验结果表明，我们的方法能有效将言语反馈应用于相关场景，同时保留其他情境下的现有行为。针对人工和GPT-4生成的高级反馈，C3PO在遵循给定反馈方面与情境基线相当，同时将过度泛化降低了30%。