Large language models (LLMs) have demonstrated remarkable performances in various tasks. However, the performance of LLMs heavily depends on the input prompt, which has given rise to a number of recent works on prompt optimization. However, previous works often require the availability of a numeric score to assess the quality of every prompt. Unfortunately, when a human user interacts with a black-box LLM, attaining such a score is often infeasible and unreliable. Instead, it is usually significantly easier and more reliable to obtain preference feedback from a human user, i.e., showing the user the responses generated from a pair of prompts and asking the user which one is preferred. Therefore, in this paper, we study the problem of prompt optimization with human feedback (POHF), in which we aim to optimize the prompt for a black-box LLM using only human preference feedback. Drawing inspiration from dueling bandits, we design a theoretically principled strategy to select a pair of prompts to query for preference feedback in every iteration, and hence introduce our algorithm named automated POHF (APOHF). We apply our APOHF algorithm to various tasks, including optimizing user instructions, prompt optimization for text-to-image generative models, and response optimization with human feedback (i.e., further refining the response using a variant of our APOHF). The results demonstrate that our APOHF can efficiently find a good prompt using a small number of preference feedback instances. Our code can be found at \url{https://github.com/xqlin98/APOHF}.

翻译：大型语言模型（LLM）已在多种任务中展现出卓越性能。然而，LLM的性能高度依赖于输入提示，这催生了近期大量关于提示优化的研究。现有方法通常需要依赖数值评分来评估每个提示的质量，但当人类用户与黑盒LLM交互时，获取此类评分往往不可行且不可靠。相比之下，从人类用户处获取偏好反馈通常更为简便可靠，即向用户展示由一对提示生成的响应，并询问用户更倾向于哪一个。因此，本文研究基于人类反馈的提示优化问题，旨在仅利用人类偏好反馈来优化黑盒LLM的提示。受对决赌博机理论的启发，我们设计了一种理论完备的策略，在每次迭代中选择一对提示以查询偏好反馈，从而提出了名为自动化POHF的算法。我们将APOHF算法应用于多项任务，包括优化用户指令、文本到图像生成模型的提示优化，以及基于人类反馈的响应优化。实验结果表明，APOHF能够通过少量偏好反馈样本高效找到优质提示。相关代码已发布于\url{https://github.com/xqlin98/APOHF}。