Large language models (LLMs) have shown impressive success in various applications. However, these models are often not well aligned with human intents, which calls for additional treatments on them, that is, the alignment problem. To make LLMs better follow user instructions, existing alignment methods mostly focus on further training them. However, the extra training of LLMs are usually expensive in terms of GPU compute; worse still, LLMs of interest are oftentimes not accessible for user-demanded training, such as GPTs. In this work, we take a different perspective -- Black-Box Prompt Optimization (BPO) -- to perform alignments. The idea is to optimize user prompts to suit LLMs' input understanding, so as to best realize users' intents without updating LLMs' parameters. BPO is model-agnostic and the empirical results demonstrate that the BPO-aligned ChatGPT yields a 22\% increase in the win rate against its original version, and 10\% for GPT-4. Importantly, the \model-aligned LLMs can outperform the same models aligned by PPO and DPO, and it also brings additional performance gains when combining \model with PPO or DPO. Code and datasets are released at https://github.com/thu-coai/BPO.

翻译：大语言模型（LLMs）在各类应用中展现出显著成功。然而，这些模型往往与人类意图不一致，这要求对其采取额外处理，即对齐问题。为使LLMs更好地遵循用户指令，现有对齐方法主要聚焦于进一步训练模型。但LLMs的额外训练通常在GPU计算方面成本高昂；更糟的是，用户关注的LLMs（如GPTs）通常无法进行用户要求的训练。本研究采用不同视角——黑盒提示优化（BPO）——来实现对齐。其核心思想是优化用户提示以适配LLMs的输入理解，从而在无需更新LLMs参数的情况下最大程度实现用户意图。BPO具有模型无关性，实证结果表明，经BPO对齐的ChatGPT相比原始版本胜率提升22%，GPT-4胜率提升10%。重要的是，经BPO对齐的LLMs性能优于经PPO和DPO对齐的相同模型，且将BPO与PPO或DPO结合使用时能带来额外性能增益。代码和数据集已发布于https://github.com/thu-coai/BPO。