Measuring long-run LLM outcomes (user satisfaction, expert judgment, downstream KPIs) is expensive. Teams default to cheap LLM judges, but uncalibrated proxies can invert rankings entirely. Causal Judge Evaluation (CJE) makes it affordable to aim at the right target: calibrate cheap scores against a small oracle slice, then evaluate at scale with valid uncertainty. We treat surrogate validity as auditable: for each policy or deployment context, a small oracle audit tests whether the learned calibration remains mean-unbiased, turning an uncheckable identification condition into a falsifiable diagnostic. On 4,961 Chatbot Arena prompts comparing five policies with a 16x oracle/judge cost ratio, at a 5% oracle fraction CJE achieves 99% pairwise ranking accuracy at 14x lower cost; across all configurations (5-50% oracle, varying n), accuracy averages 94%. An adversarial policy fails the transport audit and is correctly flagged; in such cases CJE refuses level claims rather than reporting biased estimates. Key findings: naive confidence intervals on raw judge scores achieve 0% coverage (CJE: ~95%); importance-weighted estimators fail despite >90% effective sample size; and the Coverage-Limited Efficiency (CLE) bound and its TTC diagnostic explain why.

翻译：衡量大语言模型的长期结果（用户满意度、专家评判、下游关键绩效指标）成本高昂。研究团队通常采用廉价的大语言模型评判器，但未经校准的代理指标可能完全颠倒排名结果。因果评判评估（CJE）以可承受的成本实现正确目标：通过少量黄金标准数据对廉价评分进行校准，随后在有效不确定性下进行大规模评估。我们将替代指标的有效性视为可审计的：针对每个策略或部署场景，通过小规模黄金标准审计检验习得的校准是否保持均值无偏，从而将不可验证的识别条件转化为可证伪的诊断工具。在4,961个Chatbot Arena提示词上比较五种策略（黄金标准与评判器成本比为16:1），当黄金标准数据占比5%时，CJE以降低14倍的成本实现了99%的成对排名准确率；在所有配置条件下（黄金标准占比5%-50%，样本量可变），平均准确率达到94%。对抗性策略未能通过迁移审计并被正确标记；在此类情况下，CJE拒绝提供层级声明而非报告有偏估计。关键发现：原始评判分数的朴素置信区间覆盖率为0%（CJE：约95%）；重要性加权估计量在有效样本量大于90%时依然失效；覆盖率限制效率（CLE）边界及其TTC诊断揭示了失效原因。