Measuring long-run LLM outcomes (user satisfaction, expert judgment, downstream KPIs) is expensive. Teams default to cheap LLM judges, but uncalibrated proxies can invert rankings entirely. Causal Judge Evaluation (CJE) makes it affordable to aim at the right target: calibrate cheap scores against 5% oracle labels, then evaluate at scale with valid uncertainty. On 4,961 Arena prompts, CJE achieves 99% ranking accuracy at 14x lower cost. Key findings: naive confidence intervals on uncalibrated scores achieve 0% coverage (CJE: ~95%); importance-weighted estimators fail despite 90%+ effective sample size. We introduce the Coverage-Limited Efficiency (CLE) diagnostic explaining why. CJE combines mean-preserving calibration (AutoCal-R), weight stabilization (SIMCal-W), and bootstrap inference that propagates calibration uncertainty (OUA), grounded in semiparametric efficiency theory.

翻译：衡量大语言模型（LLM）的长期结果（用户满意度、专家评判、下游关键绩效指标）成本高昂。研究团队通常采用廉价的大语言模型评判者，但未经校准的代理指标可能完全颠倒排名结果。因果评判评估（CJE）使得瞄准正确目标变得经济可行：首先利用5%的黄金标准标签对廉价评分进行校准，随后通过有效的不确定性度量进行大规模评估。在4,961个Arena提示数据集上，CJE以降低14倍的成本实现了99%的排名准确率。关键发现：基于未校准评分的朴素置信区间覆盖率为0%（CJE：约95%）；重要性加权估计量在有效样本量超过90%的情况下依然失效。我们提出了覆盖率受限效率（CLE）诊断指标以解释此现象。CJE融合了均值保持校准（AutoCal-R）、权重稳定化（SIMCal-W）以及能够传递校准不确定性的自助法推断（OUA），其理论基础来源于半参数效率理论。