LLMs are highly sensitive to prompt design, but handcrafting effective prompts is difficult and often requires intricate crafting of few-shot examples. We propose a fast automatic prompt construction algorithm that augments human instructions by generating a small set of few shot examples. Our method iteratively replaces/drops/keeps few-shot examples using Monte Carlo Shapley estimation of example utility. For faster execution, we use aggressive subsampling and a replay buffer for faster evaluations. Our method can be run using different compute time budgets. On a limited budget, we outperform existing automatic prompting methods on text simplification and GSM8K and obtain second best results on classification and summarization. With an extended, but still modest compute budget we set a new state of the art among automatic prompting methods on classification, simplification and GSM8K. Our results show that carefully constructed examples, rather than exhaustive instruction search, are the dominant lever for fast and data efficient prompt engineering. Our code is available at https://github.com/Batorskq/PIAST.

翻译：大语言模型对提示设计高度敏感，但人工设计有效提示十分困难，通常需要精心构建少量示例。本文提出一种快速自动提示构建算法，通过生成少量示例来增强人工指令。该方法利用蒙特卡洛沙普利值估计示例效用，迭代执行替换/删除/保留操作。为加速执行，我们采用激进子采样和重放缓冲区以加速评估。本方法可在不同计算时间预算下运行。在有限预算下，我们在文本简化和GSM8K任务上优于现有自动提示方法，在分类和摘要任务上取得次优结果；在适度扩展的计算预算下，我们在分类、简化和GSM8K任务上实现了自动提示方法中最先进的性能。结果表明：精心构建的示例（而非穷举式指令搜索）是实现快速、数据高效提示工程的主导因素。代码已开源：https://github.com/Batorskq/PIAST。