Large language models (LLMs) have shown impressive success in various applications. However, these models are often not well aligned with human intents, which calls for additional treatments on them, that is, the alignment problem. To make LLMs better follow user instructions, existing alignment methods mostly focus on further training them. However, the extra training of LLMs are usually expensive in terms of GPU compute; worse still, LLMs of interest are oftentimes not accessible for user-demanded training, such as GPTs. In this work, we take a different perspective -- Black-Box Prompt Optimization (BPO) -- to perform alignments. The idea is to optimize user prompts to suit LLMs' input understanding, so as to best realize users' intents without updating LLMs' parameters. BPO is model-agnostic and the empirical results demonstrate that the BPO-aligned ChatGPT yields a 22% increase in the win rate against its original version, and 10% for GPT-4. Importantly, the BPO-aligned LLMs can outperform the same models aligned by PPO and DPO, and it also brings additional performance gains when combining BPO with PPO or DPO. Code and datasets are released at https://github.com/thu-coai/BPO.

翻译：大语言模型（LLMs）在各类应用中取得了令人瞩目的成功。然而，这些模型往往与人类意图无法完全对齐，因此需要对其进行额外处理，即对齐问题。为了使LLMs更好地遵循用户指令，现有的对齐方法大多侧重于对其进一步训练。但LLMs的额外训练通常需要高昂的GPU计算成本；更糟糕的是，用户通常无法访问感兴趣的LLMs（如GPTs）并对其进行所需训练。本文采用不同视角——黑盒提示优化（BPO）来实现对齐。其核心思想是优化用户提示以适配LLMs的输入理解方式，从而在无需更新LLMs参数的情况下最大程度实现用户意图。BPO具有模型无关性，实验结果表明，经BPO对齐的ChatGPT相较于原始版本的胜率提升22%，GPT-4则提升10%。重要的是，经BPO对齐的LLMs可超越通过PPO和DPO对齐的相同模型，且将BPO与PPO或DPO结合使用时还能带来额外性能增益。代码与数据集已发布于https://github.com/thu-coai/BPO。