LLM-as-a-judge has emerged as a cornerstone technique for evaluating large language models by leveraging LLM reasoning to score prompt-response pairs. Since LLM judgments are stochastic, practitioners commonly query each pair multiple times to estimate mean scores accurately. This raises a critical challenge: given a fixed computational budget $B$, how to optimally allocate queries across $K$ prompt-response pairs to minimize estimation error? % We present a principled variance-adaptive approach leveraging multi-armed bandit theory and concentration inequalities. Our method dynamically allocates queries based on estimated score variances, concentrating resources where uncertainty is highest. Further, our algorithm is shown to achieve a worst-case score-estimation error of $\tilde{O}\left(\sqrt{\frac{\sum_{i=1}^K σ_i^2}{B}}\right)$, $σ_i^2$ being the unknown score variance for pair $i \in [K]$ with near-optimal budget allocation. % Experiments on \emph{Summarize-From-Feedback} and \emph{HelpSteer2} demonstrate that our method significantly outperforms uniform allocation, reducing worst-case estimation error while maintaining identical budgets. Our work establishes a theoretical foundation for efficient LLM evaluation with practical implications for AI safety, model alignment, and automated assessment at scale.

翻译：LLM-as-a-judge已成为评估大型语言模型的核心技术，其通过利用LLM的推理能力对提示-响应对进行评分。由于LLM的评判具有随机性，实践者通常需要对每个提示-响应对进行多次查询以准确估计平均分数。这引发了一个关键挑战：在给定固定计算预算$B$的情况下，如何将查询次数最优地分配给$K$个提示-响应对，以最小化估计误差？% 我们提出了一种基于多臂老虎机理论和集中不等式的原则性方差自适应方法。该方法根据估计的分数方差动态分配查询，将资源集中在不确定性最高的区域。进一步证明，我们的算法在最坏情况下能达到$\tilde{O}\left(\sqrt{\frac{\sum_{i=1}^K σ_i^2}{B}}\right)$的分数估计误差，其中$σ_i^2$为第$i \in [K]$个提示-响应对的未知分数方差，且实现了近乎最优的预算分配。% 在\emph{Summarize-From-Feedback}和\emph{HelpSteer2}数据集上的实验表明，在相同预算下，我们的方法显著优于均匀分配策略，在降低最坏情况估计误差的同时保持预算不变。本研究为高效的大规模LLM评估奠定了理论基础，对人工智能安全、模型对齐及自动化评估具有实际意义。