Frontier models can be prompted or conditioned to do many tasks, but finding good prompts is not always easy, nor is understanding some performant prompts. We view prompting as finding the best conditioning sequence on a near-optimal sequence predictor. On numerous well-controlled experiments, we show that unintuitive optimal conditioning sequences can be better understood given the pretraining distribution, which is not usually available. Even using exhaustive search, reliably identifying optimal prompts for practical neural predictors can be surprisingly difficult. Popular prompting methods, such as using demonstrations from the targeted task, can be surprisingly suboptimal. Using the same empirical framework, we analyze optimal prompts on frontier models, revealing patterns similar to the binary examples and previous findings. Taken together, this work takes an initial step towards understanding optimal prompts, from a statistical and empirical perspective that complements research on frontier models.

翻译：前沿模型可通过提示或条件化完成多项任务，但寻找有效提示并非易事，理解某些高性能提示同样困难。我们将提示视为在近乎最优的序列预测器上寻找最佳条件化序列。通过大量严格控制的实验表明，在通常不可获取的预训练分布条件下，某些反直觉的最优条件化序列能够得到更深入的理解。即便采用穷举搜索，为实际神经预测器可靠地识别最优提示仍异常困难。流行的提示方法（如使用目标任务的示例）可能出人意料地并非最优。基于相同的实证框架，我们分析了前沿模型上的最优提示，揭示出其与二值示例及先前研究相似的规律。总体而言，本研究从统计与实证视角出发，为理解最优提示迈出了初步探索，这恰是对前沿模型研究的重要补充。