Prompt engineering is a challenging yet crucial task for optimizing the performance of large language models (LLMs). It requires complex reasoning to examine the model's errors, hypothesize what is missing or misleading in the current prompt, and communicate the task with clarity. While recent works indicate that LLMs can be meta-prompted to perform automatic prompt engineering, their potentials may not be fully untapped due to the lack of sufficient guidance to elicit complex reasoning capabilities in LLMs in the meta-prompt. In this work, we investigate the problem of "prompt engineering a prompt engineer" -- constructing a meta-prompt that more effectively guides LLMs to perform automatic prompt engineering. We introduce and analyze key components, such as a step-by-step reasoning template and context specification, which lead to improved performance. In addition, inspired by common optimization concepts such as batch size, step size and momentum, we introduce their verbalized counterparts to the meta-prompt and investigate their effects. Our final method, named PE2, finds a prompt that outperforms "let's think step by step" by 6.3% on the MultiArith dataset and 3.1% on the GSM8K dataset. To demonstrate its versatility, we apply PE2 to the Instruction Induction benchmark, a suite of counterfactual tasks, and a lengthy, real-world industrial prompt. In these settings, PE2 achieves strong performance and outperforms prior automatic prompt engineering baselines. Further, we show that PE2 makes meaningful and targeted prompt edits, amends erroneous or incomplete prompts, and presents non-trivial counterfactual reasoning abilities.

翻译：提示工程是优化大型语言模型（LLM）性能中一项充满挑战且至关重要的任务。它需要复杂的推理能力来分析模型错误、假设当前提示中缺失或误导的信息，并清晰传达任务要求。尽管近期研究表明，大型语言模型可通过元提示（meta-prompt）实现自动提示工程，但若元提示未能充分引导LLM激发其复杂推理能力，其潜力可能无法完全释放。本研究探讨“提示工程一位提示工程师”的问题——即构建能更有效引导LLM执行自动提示工程的元提示。我们引入并分析了关键组件（如逐步推理模板和上下文规范），这些组件显著提升了性能。此外，受批量大小、步长和动量等常见优化概念的启发，我们将其言语化对应物引入元提示，并研究其影响。最终方法PE2在MultiArith数据集上比“让我们逐步思考”提示方法提升6.3%，在GSM8K数据集上提升3.1%。为证明其通用性，我们将PE2应用于指令归纳基准（Instruction Induction benchmark）、一组反事实任务以及一个冗长的工业级真实提示。在这些场景中，PE2均表现出优异性能，并超越先前自动提示工程基线。进一步研究表明，PE2能进行有意义且有针对性的提示编辑，修正错误或不完整的提示，并展现出非平凡的反事实推理能力。