In many modern LLM applications, such as retrieval augmented generation, prompts have become programs themselves. In these settings, prompt programs are repeatedly called with different user queries or data instances. A big practical challenge is optimizing such prompt programs. Recent work has mostly focused on either simple prompt programs or assumed that the general structure of a prompt program is fixed. We introduce SAMMO, a framework to perform symbolic prompt program search for compile-time optimizations of prompt programs. SAMMO represents prompt programs on a symbolic level which allows for a rich set of transformations that can be searched over during optimization. We show that SAMMO generalizes previous methods and improves the performance of complex prompts on (1) instruction tuning, (2) RAG pipeline tuning, and (3) prompt compression, across several different LLMs. We make all code available open-source at https://github.com/microsoft/sammo .

翻译：在许多现代大语言模型应用中，例如检索增强生成，提示本身已成为程序。在这些场景中，提示程序会针对不同的用户查询或数据实例被反复调用。一个重大的实际挑战是优化此类提示程序。近期工作大多集中于简单的提示程序，或假设提示程序的整体结构是固定的。我们介绍了SAMMO，一个用于执行符号化提示程序搜索以实现提示程序编译时优化的框架。SAMMO在符号层面表示提示程序，这允许在优化过程中搜索一组丰富的转换操作。我们证明，SAMMO推广了先前的方法，并在多种不同的大语言模型上，针对（1）指令调优，（2）RAG管道调优，以及（3）提示压缩，提升了复杂提示的性能。我们已在 https://github.com/microsoft/sammo 开源所有代码。