Free-form legal essay evaluation in NLP treats expert inter-rater stability as a single ceiling number, and treats LLM-judge agreement with that ceiling as evidence of judge stability. We test both assumptions on the Thai bar examination through an identical-inputs protocol: three Bar Council-trained examiners (A, B, C) and a 26-LLM judge panel score the same 15 cross-graded answers from the same four inputs (question, official Bar Council grading regulation, gold answer, candidate answer). The headline finding is asymmetric. On 10 of 15 cells where the rubric prescribes both axes, all 29 raters converge in a tight band: panel agreement is universal. On the remaining 5 cells where the rubric does not prescribe how to grade a correct final answer that omits a decisive statutory citation, the human panel splits between two coherent readings (B/C majority at the upper rubric band, score 6-8; A minority at the lower band, score 1-2). The LLM judge population does not split symmetrically: 22 of 26 LLMs score in or near B/C's contested band, 3 sit in the regulation-silent middle gap, and only 1 (GPT-5.4 Nano) approaches A's band without consistently scoring within it. Zero LLMs in our 26-judge panel reproduce the minority human reading on the contested cells. The B/C-direction cluster spans every model size, vendor, and price tier we tested. An instrumented three-LLM anchor sub-panel (Claude 4.6 Opus, Gemini 3.1 Pro, GPT-5.4 Pro) carries determinism probes, input ablations, and bootstrap CIs, and reaches anchor panel $α= 0.77$ on the 15 cells against human-panel $α= 0.36$. The high LLM-panel $α$ reflects systematic convergence on the majority reading rather than balanced reproduction of both readings; a benchmark that selects its LLM judge by maximising agreement with a human reference panel will inherit this asymmetry by construction.

翻译：自然语言处理中自由形式法律论文评估将专家间评分稳定性视为单一上限数值，并将LLM评判官与该上限的一致性视为评判稳定的证据。我们通过相同输入协议在泰国律师资格考试中检验这两个假设：三位律师委员会培训的审查员（A、B、C）与26名LLM评判官小组依据相同四个输入（试题、官方律师委员会评分规则、参考答案、考生答案）对同一15道交叉评阅答卷进行评分。主要发现呈现非对称性：在评分规则明确指定两个评分维度的10个案例中，全部29位评分者评分高度集中——小组一致性普遍存在；而在其余5个评分规则未明确规定如何评分（即正确答案虽正确但省略决定性法律条款引用）的案例中，人类评审小组分裂为两种相互矛盾的合理解读（B/C多数派对应评分规则上限区间，得分6-8分；A少数派对应下限区间，得分1-2分）。LLM评判官群体并未呈现对称分裂：26个LLM中有22个评分落在或接近B/C派别争议区间，3个位于评分规则未明确的中间空白区域，仅1个（GPT-5.4 Nano）接近A派别区间但未持续落在该区间内。我们26人评判小组中没有任何LLM在争议案例中重现少数派人类解读。B/C方向聚类覆盖我们测试的所有模型规模、厂商及价格层级。由三个LLM组成的标定小组（Claude 4.6 Opus、Gemini 3.1 Pro、GPT-5.4 Pro）通过确定性探测、输入消融实验及自助法置信区间分析，其15个案例的评分者间信度系数α=0.77，而人类小组α=0.36。LLM小组的高α值反映的是系统性收敛于多数派解读，而非平衡重现两种解读；若基准测试通过最大化与人类参考小组的一致性来选择LLM评判官，则该基准在结构上必然继承这种非对称性。