Large Language Models (LLMs) exhibit strong generalization capabilities to novel tasks when prompted with language instructions and in-context demos. Since this ability sensitively depends on the quality of prompts, various methods have been explored to automate the instruction design. While these methods demonstrated promising results, they also restricted the searched prompt to one instruction. Such simplification significantly limits their capacity, as a single demo-free instruction might not be able to cover the entire complex problem space of the targeted task. To alleviate this issue, we adopt the Mixture-of-Expert paradigm and divide the problem space into a set of sub-regions; Each sub-region is governed by a specialized expert, equipped with both an instruction and a set of demos. A two-phase process is developed to construct the specialized expert for each region: (1) demo assignment: Inspired by the theoretical connection between in-context learning and kernel regression, we group demos into experts based on their semantic similarity; (2) instruction assignment: A region-based joint search of an instruction per expert complements the demos assigned to it, yielding a synergistic effect. The resulting method, codenamed Mixture-of-Prompts (MoP), achieves an average win rate of 81% against prior arts across several major benchmarks.

翻译：大型语言模型（LLMs）在接收语言指令和上下文示例提示时，展现出对新颖任务的强大泛化能力。由于这种能力高度依赖于提示的质量，研究者已探索多种方法来自动化指令设计。尽管这些方法取得了有前景的结果，但它们通常将搜索的提示限制在单一指令上。这种简化显著限制了其能力，因为单个无示例的指令可能无法覆盖目标任务所对应的整个复杂问题空间。为缓解这一问题，我们采用专家混合范式，将问题空间划分为若干子区域；每个子区域由一位配备专用指令和一组示例的专家负责管理。我们开发了一个两阶段流程来为每个区域构建专用专家：（1）示例分配：受上下文学习与核回归理论关联的启发，我们根据示例的语义相似性将其分组至不同专家；（2）指令分配：基于区域的联合搜索为每位专家匹配与其分配示例互补的指令，从而产生协同效应。该方法代号为“混合提示”（Mixture-of-Prompts, MoP），在多个主流基准测试中相对现有方法平均胜率达到81%。