Prompt optimization methods have demonstrated significant effectiveness in aligning black-box large language models (LLMs). In parallel, inference scaling strategies such as Best-of-N Sampling and Majority Voting have likewise been shown to improve alignment and performance by trading additional computation for better output. However, existing prompt optimization approaches are inference strategy agnostic; that is, they optimize prompts without accounting for the inference strategy. This constitutes a significant methodological gap, as our empirical and theoretical analysis reveals a strong interdependence between these two paradigms. Moreover, we find that user preferences regarding trade-offs among multiple objectives and inference budgets substantially influence the choice of prompt and inference configuration. To address this gap, we introduce a novel unified framework named IAPO (Inference-Aware Prompt Optimization) that jointly optimizes the prompt and inference scale, while being aware of the inference budget and different task objectives. We then develop a fixed-budget training algorithm for IAPO, called PSST (Prompt Scaling via Sequential Trimming), and establish finite-budget guarantees on the error probability. Finally, we evaluate the effectiveness of PSST on six tasks, including multi-objective text generation and reasoning, and demonstrate the critical role of incorporating inference-awareness in aligning black-box LLMs using prompt optimization.

翻译：提示优化方法在实现对黑盒大语言模型（LLMs）的对齐方面已展现出显著的有效性。与此同时，推理扩展策略，如最佳N采样（Best-of-N Sampling）和多数投票（Majority Voting），同样被证明可以通过牺牲额外计算来换取更优输出，从而改善对齐效果和性能。然而，现有的提示优化方法独立于推理策略；即它们在优化提示时未考虑推理策略。这构成了一个重要的方法论缺陷，因为我们的实证与理论分析揭示了这两种范式之间存在强烈的相互依赖关系。此外，我们发现，用户对于多个目标与推理预算之间权衡的偏好，显著影响着提示和推理配置的选择。为弥补这一缺陷，我们引入了一个名为IAPO（推理感知提示优化）的新型统一框架，该框架在感知推理预算和不同任务目标的同时，联合优化提示和推理规模。随后，我们为IAPO开发了一种固定预算训练算法，称为PSST（通过顺序剪枝的提示扩展），并建立了关于错误概率的有限预算保证。最后，我们在包括多目标文本生成和推理在内的六项任务上评估了PSST的有效性，并证明了在利用提示优化对齐黑盒LLMs时，纳入推理感知的关键作用。