Large language models (LLMs) are increasingly used as automated evaluators, yet prior works demonstrate that these LLM judges often lack consistency in scoring when the prompt is altered. However, the effect of the grading scale itself remains underexplored. We study the LLM-as-a-judge problem by comparing two kinds of raters: humans and LLMs. We collect ratings from both groups on three scales and across six benchmarks that include objective, open-ended subjective, and mixed tasks. Using intraclass correlation coefficients (ICC) to measure absolute agreement, we find that LLM judgments are not perfectly consistent across scales on subjective benchmarks, and that the choice of scale substantially shifts human-LLM agreement, even when within-group panel reliability is high. Aggregated over tasks, the grading scale of 0-5 yields the strongest human-LLM alignment. We further demonstrate that pooled reliability can mask benchmark heterogeneity and reveal systematic subgroup differences in alignment across gender groups, strengthening the importance of scale design and sub-level diagnostics as essential components of LLM-as-a-judge protocols.

翻译：大型语言模型（LLM）正日益被用作自动化评估工具，然而先前的研究表明，当提示信息改变时，这些LLM评判者在评分中往往缺乏一致性。然而，评分尺度本身的影响仍未得到充分探索。我们通过比较两类评分者——人类与LLM——来研究LLM作为评判者的问题。我们在三种评分尺度上收集了来自两个群体的评分数据，涵盖六个基准测试，包括客观任务、开放式主观任务以及混合任务。使用组内相关系数（ICC）来衡量绝对一致性，我们发现LLM的评判在主观基准测试中并非完全跨尺度一致，并且尺度的选择会显著改变人类与LLM之间的一致性，即使在组内评审团信度较高的情况下也是如此。综合各项任务来看，0-5分的评分尺度产生了最强的人类-LLM一致性。我们进一步证明，汇总的信度可能掩盖基准测试的异质性，并揭示了不同性别群体间在一致性上存在系统性亚组差异，这强化了尺度设计和次级诊断作为LLM作为评判者协议中不可或缺组成部分的重要性。