While large language models (LLMs) are increasingly used as automatic judges for question answering (QA) and other reference-conditioned evaluation tasks, little is known about their ability to adhere to a provided reference. We identify a critical failure mode of such reference-based LLM QA evaluation: when the provided reference conflicts with the judge model's parametric knowledge, the resulting scores become unreliable, substantially degrading evaluation fidelity. To study this phenomenon systematically, we introduce a controlled swapped-reference QA framework that induces reference-belief conflicts. Specifically, we replace the reference answer with an incorrect entity and construct diverse pairings of original and swapped references with correspondingly aligned candidate answers. Surprisingly, grading reliability drops sharply under swapped references across a broad set of judge models. We empirically show that this vulnerability is driven by judges' over-reliance on parametric knowledge, leading judges to disregard the given reference under conflict. Finally, we find that this failure persists under common prompt-based mitigation strategies, highlighting a fundamental limitation of LLM-as-a-judge evaluation and motivating reference-based protocols that enforce stronger adherence to the provided reference.

翻译：尽管大型语言模型（LLM）越来越多地被用作问答（QA）及其他基于参考的评估任务的自动评判者，但其遵循给定参考的能力尚未得到充分研究。我们发现此类基于参考的LLM问答评估存在一个关键失效模式：当提供的参考与评判模型的参数化知识发生冲突时，生成的评分将变得不可靠，从而显著降低评估的保真度。为系统研究这一现象，我们引入一种受控的置换参考问答框架，用以诱发参考-信念冲突。具体而言，我们将原始参考答案替换为错误实体，并构建原始参考与置换参考的多样化配对及其对应对齐的候选答案。令人惊讶的是，在广泛的评判模型集合中，置换参考条件下的评分可靠性均急剧下降。我们通过实证表明，该脆弱性源于评判者对参数化知识的过度依赖，导致其在冲突情况下忽视给定的参考。最后，我们发现该失效现象在常见的基于提示的缓解策略下依然存在，这揭示了LLM作为评判者的评估方法存在根本性局限，并促使我们探索能够强制模型更严格遵循给定参考的基于参考的评估协议。