The paradigm of LLM-as-a-judge relies on a critical assumption, namely that high inter-evaluator agreement indicates reliable and objective evaluation. We present two complementary findings that challenge this assumption. \textbf{First}, we demonstrate that this consensus is frequently illusory. We identify and formalize \textbf{Evaluation Illusion}, a phenomenon where LLM judges generate sophisticated critiques yet anchor scores on shared surface heuristics rather than substantive quality. Through a large-scale study of 105,600 evaluation instances (32 LLMs $\times$ 3 frontier judges $\times$ 100 tasks $\times$ 11 temperatures), we show that model-level agreement (Spearman $ρ= 0.99$) masks fragile sample-level agreement (Pearson $\bar{r} = 0.72$; absolute agreement ICC $= 0.67$), that merely sharing rubric structure restores 62\% of total agreement, and that high-quality outputs paradoxically receive the \textit{least} consistent evaluations. \textbf{Second}, we demonstrate that dynamically generating evaluation rubrics grounded in domain knowledge produces more meaningful assessment. We introduce MERG (Metacognitive Enhanced Rubric Generation), a knowledge-driven rubric generation framework whose domain-selective effects confirm this. Agreement \textit{increases} in codified domains (Education +22\%, Academic +27\%) where knowledge anchors evaluators on shared standards, while it decreases in subjective domains where genuine evaluative pluralism emerges. These findings suggest that evaluation rubrics should be dynamically enriched with expert knowledge rather than relying on generic criteria, with implications for reward modeling in RLAIF.

翻译：LLM作为评判者的范式依赖于一个关键假设，即高评估者间一致性意味着可靠且客观的评估。我们提出了两个互补的发现，挑战了这一假设。**首先**，我们证明这种共识常常是虚幻的。我们识别并形式化了**评估幻象**，即LLM评判者生成复杂的批评，却将评分锚定在共享的表面启发式而非实质质量上。通过对105,600个评估实例（32个LLM × 3个前沿评判者 × 100个任务 × 11个温度）的大规模研究，我们表明模型层面的一致性（Spearman $ρ= 0.99$）掩盖了脆弱的样本层面一致性（Pearson $\bar{r} = 0.72$；绝对一致性 ICC $= 0.67$），仅仅共享评估准则结构就能恢复总一致性的62%，并且高质量输出反而获得**最不一致**的评估。**其次**，我们证明基于领域知识动态生成评估准则能产生更有意义的评估。我们引入了MERG（元认知增强准则生成），这是一个知识驱动的准则生成框架，其领域选择性效应证实了这一点。在知识能将评估者锚定于共享标准的编码化领域（教育+22%，学术+27%），一致性**增加**；而在主观领域，一致性下降，真正的评估多元性得以显现。这些发现表明，评估准则应动态地融入专家知识，而非依赖通用标准，这对RLAIF中的奖励建模具有启示意义。