Rigorous evaluation of large language models (LLMs) relies on comparing models by the prevalence of desirable or undesirable behaviors, such as task pass rates or policy violations. These prevalence estimates are produced by a classifier, either an LLM-as-a-judge or human annotators, making the choice of classifier central to trustworthy evaluation. Common metrics used for this choice, such as Accuracy, Precision, and F1, are sensitive to class imbalance and to arbitrary choices of positive class, and can favor judges that distort prevalence estimates. We show that Youden's $J$ statistic is theoretically aligned with choosing the best judge to compare models, and that Balanced Accuracy is an equivalent linear transformation of $J$. Through both analytical arguments and empirical examples and simulations, we demonstrate how selecting judges using Balanced Accuracy leads to better, more robust classifier selection.

翻译：大型语言模型（LLM）的严谨评估依赖于通过期望或不期望行为的普遍性来比较模型，例如任务通过率或策略违规情况。这些普遍性估计由分类器（无论是LLM作为评判者还是人工标注者）产生，使得分类器的选择成为可信评估的核心。用于此选择的常见指标，如准确率、精确率和F1分数，对类别不平衡和正类的任意选择敏感，并且可能偏向于扭曲普遍性估计的评判者。我们证明尤登$J$统计量在理论上与选择最佳评判者以比较模型的目标一致，且平衡准确率是$J$的等效线性变换。通过理论分析和实证案例及模拟，我们展示了使用平衡准确率选择评判者如何带来更好、更稳健的分类器选择。