The efficacy of large language models (LLMs) in understanding and generating natural language has aroused a wide interest in developing prompt-based methods to harness the power of black-box LLMs. Existing methodologies usually prioritize a global optimization for finding the global optimum, which however will perform poorly in certain tasks. This thus motivates us to re-think the necessity of finding a global optimum in prompt optimization. To answer this, we conduct a thorough empirical study on prompt optimization and draw two major insights. Contrasting with the rarity of global optimum, local optima are usually prevalent and well-performed, which can be more worthwhile for efficient prompt optimization (Insight I). The choice of the input domain, covering both the generation and the representation of prompts, affects the identification of well-performing local optima (Insight II). Inspired by these insights, we propose a novel algorithm, namely localized zeroth-order prompt optimization (ZOPO), which incorporates a Neural Tangent Kernel-based derived Gaussian process into standard zeroth-order optimization for an efficient search of well-performing local optima in prompt optimization. Remarkably, ZOPO outperforms existing baselines in terms of both the optimization performance and the query efficiency, which we demonstrate through extensive experiments.

翻译：大型语言模型（LLMs）在自然语言理解与生成中的有效性，催生了利用黑盒LLMs能力的提示方法的广泛研究。现有方法通常优先进行全局优化以寻找全局最优解，但在特定任务中往往表现不佳。这促使我们重新思考提示优化中寻找全局最优解的必要性。为此，本文对提示优化展开深入实证研究，得出两个重要见解：与全局最优解的稀缺性形成对比，局部最优解通常普遍存在且表现优异，这对高效提示优化更具价值（见解I）；输入域的选择（涵盖提示生成与表示）会影响优质局部最优解的识别（见解II）。受这些见解启发，我们提出一种新型算法——局部零阶提示优化（ZOPO），该算法将基于神经正切核的高斯过程融入标准零阶优化，以实现提示优化中高性能局部最优解的高效搜索。值得注意的是，ZOPO在优化性能与查询效率两方面均超越了现有基线方法，这一点已通过大量实验验证。