Confidence estimation infers a probability for whether each model output is correct or not. While predicting such binary correctness is sensible for tasks with exact answers, free-form generation tasks are often more nuanced, with output quality being both fine-grained and multi-faceted. We thus propose Performance Interval Estimation (PIE) to predict both: 1) point estimates for any arbitrary set of continuous-valued evaluation metrics; and 2) calibrated uncertainty intervals around these point estimates. We then compare two approaches: LLM-as-judge vs. classic regression with confidence estimation features. Evaluation over 11 datasets spans summarization, translation, code generation, function-calling, and question answering. Regression is seen to achieve both: i) lower error point estimates of metric scores; and ii) well-calibrated uncertainty intervals. To support reproduction and follow-on work, we share our data and code.

翻译：置信度估计旨在推断每个模型输出正确与否的概率。对于具有确切答案的任务而言，预测此类二元正确性是合理的，但自由形式生成任务通常更为微妙，其输出质量既是细粒度的，也是多方面的。因此，我们提出性能区间估计（PIE）来同时预测：1）针对任意连续值评估指标集合的点估计；以及2）围绕这些点估计的校准不确定性区间。随后，我们比较了两种方法：LLM作为评判器与带有置信度估计特征的传统回归方法。在涵盖摘要、翻译、代码生成、函数调用和问答的11个数据集上的评估表明，回归方法能够同时实现：i）更低的指标得分点估计误差；以及ii）良好校准的不确定性区间。为支持复现和后续研究，我们公开了相关数据与代码。