Free-form legal essay evaluation in NLP treats expert inter-rater stability as a single ceiling number, and treats LLM-judge agreement with that ceiling as evidence of judge stability. We test both assumptions on the Thai bar examination through an identical-inputs protocol: three Bar Council-trained examiners (A, B, C) and a 26-LLM judge panel score the same 15 cross-graded answers from the same four inputs (question, official Bar Council grading regulation, gold answer, candidate answer). The headline finding is asymmetric. On 10 of 15 cells where the rubric prescribes both axes, all 29 raters converge in a tight band: panel agreement is universal. On the remaining 5 cells where the rubric does not prescribe how to grade a correct final answer that omits a decisive statutory citation, the human panel splits between two coherent readings (B/C majority at the upper rubric band, score $6$--$8$; A minority at the lower band, score $1$--$2$). The LLM judge population does not split symmetrically: 22 of 26 LLMs score in or near B/C's contested band, 3 sit in the regulation-silent middle gap, and only 1 (GPT-5.4 Nano) approaches A's band without consistently scoring within it. \emph{Zero LLMs in our 26-judge panel reproduce the minority human reading on the contested cells.} The B/C-direction cluster spans every model size, vendor, and price tier we tested. An instrumented three-LLM anchor sub-panel (Claude 4.6 Opus, Gemini 3.1 Pro, GPT-5.4 Pro) carries determinism probes, input ablations, and bootstrap CIs, and reaches anchor panel $α= 0.77$ on the 15 cells against human-panel $α= 0.36$. The high LLM-panel $α$ reflects systematic convergence on the majority reading rather than balanced reproduction of both readings; a benchmark that selects its LLM judge by maximising agreement with a human reference panel will inherit this asymmetry by construction.

翻译：自然语言处理中自由格式法律论文评估将专家间评分者间信度视为单一上限值，并将LLM评委与该上限的一致性视为评委稳定性的证据。我们通过相同输入协议在泰国律师资格考试中检验这两个假设：三名律师委员会培训考官（A、B、C）与包含26个LLM的评委小组，基于相同的四个输入（考题、官方律师委员会评分规则、标准答案、考生答案）对15道交叉评阅的考题进行评分。主要发现呈现非对称性。在评分标准同时规定两个评分维度的15个单元格中，所有29位评分者高度集中：评审小组达成普遍一致。而在剩余5个未规定对省略关键法条引用的正确最终答案如何评分的单元格中，人类评审小组分裂为两种合理解释（B/C组多数采用评分标准上限，得分6\~8分；A组少数采用下限，得分1\~2分）。LLM评委群体并未对称分裂：26个LLM中22个评分落在或接近B/C组争议区间，3个位于规则未覆盖的中部空白区，仅1个（GPT-5.4 Nano）接近A组评分区间但未持续落入该区间。\emph{在26个LLM评委中，没有任何模型在争议单元格上复现少数人类评审的解读。}B/C方向聚类涵盖了我们测试的所有模型规模、供应商和价格层级。由三个LLM锚定（Claude 4.6 Opus、Gemini 3.1 Pro、GPT-5.4 Pro）组成的工具化子小组进行了确定性探测、输入消融实验和自助法置信区间分析，在15个单元格上锚定小组内部α系数为0.77，而人类小组α系数为0.36。LLM小组高α系数反映的是对多数解读的系统性趋同，而非对两种解读的均衡复现；若基准测试通过最大化与人类参考小组的一致性来选择其LLM评委，则系统将继承这种非对称性。