LLMs are highly sensitive to prompt design, but handcrafting effective prompts is difficult and often requires intricate crafting of few-shot examples. We propose a fast automatic prompt construction algorithm that augments human instructions by generating a small set of few shot examples. Our method iteratively replaces/drops/keeps few-shot examples using Monte Carlo Shapley estimation of example utility. For faster execution, we use aggressive subsampling and a replay buffer for faster evaluations. Our method can be run using different compute time budgets. On a limited budget, we outperform existing automatic prompting methods on text simplification and GSM8K and obtain second best results on classification and summarization. With an extended, but still modest compute budget we set a new state of the art among automatic prompting methods on classification, simplification and GSM8K. Our results show that carefully constructed examples, rather than exhaustive instruction search, are the dominant lever for fast and data efficient prompt engineering. Our code is available at https://github.com/Batorskq/PIAST.

翻译：大语言模型（LLMs）对提示设计高度敏感，但手工构建有效提示十分困难，且通常需要精心设计少样本示例。本文提出一种快速自动提示构建算法，通过生成少量少样本示例来增强人工指令。该方法利用蒙特卡洛夏普利估计（Monte Carlo Shapley estimation）评估示例效用，迭代地替换/丢弃/保留少样本示例。为提升执行速度，我们采用激进子采样与回放缓冲区加速评估。该方法可在不同计算时间预算下运行。在有限预算下，本方法在文本简化和GSM8K任务上优于现有自动提示方法，在分类与摘要任务上取得次优结果。当采用扩展但仍适度的计算预算时，本方法在分类、文本简化和GSM8K任务上均创下自动提示方法的最新最优结果。实验证明，精心构建的示例（而非穷举式指令搜索）是实现快速且数据高效的提示工程的关键因素。本方法代码开源于 https://github.com/Batorskq/PIAST。