In this work, we address the problem of directing the text generations of a LLM towards a desired behavior, aligning the generated text with the preferences of the human operator. We propose using another language model as a critic, reward model in a zero-shot way thanks to the prompt of a Yes-No question that represents the user preferences, without requiring further labeled data. This zero-shot reward model provides the learning signal to further fine-tune the base LLM using reinforcement learning, as in RLAIF; yet our approach is also compatible in other contexts such as quality-diversity search. Extensive evidence of the capabilities of the proposed ZYN framework is provided through experiments in different domains related to text generation, including detoxification; optimizing sentiment of movie reviews, or any other attribute; steering the opinion about a particular topic the model may have; and personalizing prompt generators for text-to-image tasks. Code to be released at \url{https://github.com/vicgalle/zero-shot-reward-models/}.

翻译：本文研究如何引导大型语言模型的文本生成朝向期望行为，使生成文本与人类操作者的偏好对齐。我们提出利用另一语言模型作为评论者，通过代表用户偏好的是非问题提示，以零样本方式构建奖励模型，无需额外标注数据。该零样本奖励模型可为基于强化学习的基座大模型微调（如RLAIF方法）提供学习信号；同时，该方法也兼容质量-多样性搜索等其他应用场景。通过在文本生成相关领域的多维度实验，包括去毒化、电影评论情感优化及其他属性调控、特定话题观点导向修正、以及文本到图像任务中提示生成器的个性化定制，我们充分验证了所提ZYN框架的有效性。代码将于\url{https://github.com/vicgalle/zero-shot-reward-models/}发布。