Prompt-based learning has been an effective paradigm for large pretrained language models (LLM), enabling few-shot or even zero-shot learning. Black-box prompt search has received growing interest recently for its distinctive properties of gradient-free optimization, proven particularly useful and powerful for model-as-a-service usage. However, the discrete nature and the complexity of combinatorial optimization hinder the efficiency of modern black-box approaches. Despite extensive research on search algorithms, the crucial aspect of search space design and optimization has been largely overlooked. In this paper, we first conduct a sensitivity analysis by prompting LLM, revealing that only a small number of tokens exert a disproportionate amount of influence on LLM predictions. Leveraging this insight, we propose the Clustering and Pruning for Efficient Black-box Prompt Search (ClaPS), a simple black-box search method that first clusters and prunes the search space to focus exclusively on influential prompt tokens. By employing even simple search methods within the pruned search space, ClaPS achieves state-of-the-art performance across various tasks and LLMs, surpassing the performance of complex approaches while significantly reducing search costs. Our findings underscore the critical role of search space design and optimization in enhancing both the usefulness and the efficiency of black-box prompt-based learning.

翻译：提示学习已成为大型预训练语言模型的有效范式，能够实现少样本甚至零样本学习。近年来，黑盒提示搜索因其独特的无梯度优化特性而获得广泛关注，尤其在模型即服务场景中展现出实用性与强大能力。然而，离散性质与组合优化的复杂性阻碍了现代黑盒方法的效率。尽管搜索算法研究已取得大量进展，但搜索空间设计与优化这一关键环节却长期被忽视。本文首先通过对大语言模型进行提示敏感性分析，发现仅有少量token对模型预测产生显著影响。基于此洞察，我们提出基于聚类与剪枝的高效黑盒提示搜索方法，该方法通过聚类与剪枝搜索空间，聚焦于有影响力的提示token。即使在剪枝后的搜索空间中采用简单搜索方法，ClaPS方法在不同任务与大语言模型上均能实现最优性能，在显著降低搜索成本的同时超越复杂方法的表现。我们的研究结果凸显了搜索空间设计与优化在提升黑盒提示学习的实用性与效率中的关键作用。