Existing methods for controlling language models, such as RLHF and Constitutional AI, involve determining which LLM behaviors are desirable and training them into a language model. However, in many cases, it is desirable for LLMs to be controllable at inference time, so that they can be used in multiple contexts with diverse needs. We illustrate this with the Pink Elephant Problem: instructing an LLM to avoid discussing a certain entity (a ``Pink Elephant''), and instead discuss a preferred entity (``Grey Elephant''). We apply a novel simplification of Constitutional AI, Direct Principle Feedback, which skips the ranking of responses and uses DPO directly on critiques and revisions. Our results show that after DPF fine-tuning on our synthetic Pink Elephants dataset, our 13B fine-tuned LLaMA 2 model significantly outperforms Llama-2-13B-Chat and a prompted baseline, and performs as well as GPT-4 in on our curated test set assessing the Pink Elephant Problem.

翻译：现有控制语言模型的方法（如RLHF和Constitutional AI）通常需要确定LLM的期望行为并将其训练到模型中。然而在许多场景中，希望LLM在推理阶段具有可控性，以便能适用于不同需求的多种应用场景。我们通过"粉红大象问题"阐释这一需求：要求LLM避免讨论某个特定实体（"粉红大象"），转而讨论另一个优选实体（"灰色大象"）。我们提出Constitutional AI的一种新颖简化方案——直接原则反馈(DPF)，该方法跳过响应的排序步骤，直接对批评和修正结果使用DPO算法。实验结果表明，在合成粉红大象数据集上进行DPF微调后，我们的13B参数LLaMA 2精调模型显著优于Llama-2-13B-Chat模型及提示基线方法，在评估粉红大象问题的定制测试集上与GPT-4表现相当。