Large language model (LLM) prompting is a promising new approach for users to create and customize their own chatbots. However, current methods for steering a chatbot's outputs, such as prompt engineering and fine-tuning, do not support users in converting their natural feedback on the model's outputs to changes in the prompt or model. In this work, we explore how to enable users to interactively refine model outputs through their feedback, by helping them convert their feedback into a set of principles (i.e. a constitution) that dictate the model's behavior. From a formative study, we (1) found that users needed support converting their feedback into principles for the chatbot and (2) classified the different principle types desired by users. Inspired by these findings, we developed ConstitutionMaker, an interactive tool for converting user feedback into principles, to steer LLM-based chatbots. With ConstitutionMaker, users can provide either positive or negative feedback in natural language, select auto-generated feedback, or rewrite the chatbot's response; each mode of feedback automatically generates a principle that is inserted into the chatbot's prompt. In a user study with 14 participants, we compare ConstitutionMaker to an ablated version, where users write their own principles. With ConstitutionMaker, participants felt that their principles could better guide the chatbot, that they could more easily convert their feedback into principles, and that they could write principles more efficiently, with less mental demand. ConstitutionMaker helped users identify ways to improve the chatbot, formulate their intuitive responses to the model into feedback, and convert this feedback into specific and clear principles. Together, these findings inform future tools that support the interactive critiquing of LLM outputs.

翻译：大型语言模型（LLM）提示是用户创建和定制自己聊天机器人的一种有前景的新方法。然而，当前引导聊天机器人输出的方法（如提示工程和微调）无法支持用户将对模型输出的自然反馈转化为提示或模型的修改。在本研究中，我们探索如何通过帮助用户将反馈转化为一组约束模型行为的准则（即宪法），从而使用户能够通过其反馈交互式地优化模型输出。通过一项形成性研究，我们（1）发现用户需要支持以将其反馈转化为聊天机器人的准则，（2）分类了用户所需的不同准则类型。受这些发现启发，我们开发了ConstitutionMaker——一种将用户反馈转化为准则以引导基于LLM聊天机器人的交互式工具。使用ConstitutionMaker，用户可以用自然语言提供正面或负面反馈、选择自动生成的反馈或重写聊天机器人的回复；每种反馈模式会自动生成一条准则，并插入到聊天机器人的提示中。在包含14名参与者的用户研究中，我们将ConstitutionMaker与消融版本（用户自行编写准则）进行对比。使用ConstitutionMaker时，参与者认为其准则能更好地引导聊天机器人，他们能更轻松地将反馈转化为准则，且能以更低的脑力需求更高效地编写准则。ConstitutionMaker帮助用户识别改进聊天机器人的方法，将他们对模型的直觉反应转化为反馈，并将这些反馈转化为具体清晰的准则。这些发现共同为未来支持对LLM输出进行交互式批评的工具提供了设计依据。