We consider the sequential experimental design problem in the predict-then-optimize paradigm. In this paradigm, the outputs of the prediction model are used as coefficient vectors in a downstream linear optimization problem. Traditional sequential experimental design aims to control the input variables (features) so that the improvement in prediction accuracy from each experimental outcome (label) is maximized. However, in the predict-then-optimize setting, performance is ultimately evaluated based on the decision loss induced by the downstream optimization, rather than by prediction error. This mismatch between prediction accuracy and decision loss renders traditional decision-blind designs inefficient. To address this issue, we propose a directional-based metric to quantify predictive uncertainty. This metric does not require solving an optimization oracle and is therefore computationally tractable. We show that the resulting sequential design criterion enjoys strong consistency and convergence guarantees. Under a broad class of distributions, we demonstrate that our directional uncertainty-based design attains an earlier stopping time than decision-blind designs. This advantage is further supported by real-world experiments on an LLM job allocation problem.

翻译：我们考虑预测-优化范式中的序列实验设计问题。在该范式中，预测模型的输出被用作下游线性优化问题中的系数向量。传统序列实验设计旨在控制输入变量（特征），使得每次实验结果（标签）带来的预测精度提升最大化。然而在预测-优化场景中，性能最终基于下游优化引发的决策损失进行评估，而非预测误差。预测精度与决策损失之间的不匹配导致传统无视决策的设计效率低下。为解决该问题，我们提出一种基于方向性的度量来量化预测不确定性。该度量无需调用优化求解器，因而具有计算可操作性。我们证明所得序列设计准则具有强一致性与收敛性保证。在广泛分布的类别下，我们证明基于方向性不确定性的设计比无视决策的设计能获得更早的停止时间。这一优势在LLM任务分配问题的实际实验中得到了进一步验证。