Recent research has shown that large language models (LLMs) favor their own outputs when acting as judges, undermining the integrity of automated post-training and evaluation workflows. However, it is difficult to disentangle which evaluation biases are explained by narcissism versus general experimental confounds, distorting measurements of self-preference bias. We discover a core methodological confound which could reduce measurement error by 89.6%. Specifically, LLM evaluators may deliver self-preferring verdicts when the judge responds to queries which they completed incorrectly themselves; this would be true regardless of whether one of their responses is their own. To decouple self-preference signals from noisy outputs on hard problems, we introduce an Evaluator Quality Baseline, which compares the probability that a judge incorrectly votes for itself against the probability that it votes for an incorrect response from another model. Evaluating this simple baseline on 37,448 queries, only 51% of initial findings retain statistical significance. Finally, we turn towards characterizing the entropy of "easy" versus "hard" evaluation votes from LLM judges. Our corrective baseline enables future research on self-preference by eliminating noisy data from potential solutions. More widely, this work contributes to the growing body of work on cataloging and isolating judge-bias effects.

翻译：近期研究表明，大语言模型在作为评估者时倾向于偏爱自身输出，这损害了自动化后训练与评估流程的完整性。然而，我们难以区分哪些评估偏差源于自恋效应，哪些源于一般性实验混杂因素，这导致对自我偏好偏差的测量产生扭曲。我们发现一个核心方法学混杂因素，其可降低89.6%的测量误差。具体而言，当评估模型对自身曾错误回答的查询进行评判时，可能会给出自我偏好的结论——即使其中某个回答并非其自身生成，这一现象仍会出现。为从困难问题产生的噪声输出中分离自我偏好信号，我们提出评估者质量基线方法，通过比较评估者错误选择自身回答的概率与选择其他模型错误回答的概率来实现解耦。在37,448条查询数据上对该简易基线进行评估后，仅51%的初始发现保持统计显著性。最后，我们转向刻画大语言模型评估者对"简单"与"困难"评估投票的熵值特征。本校正基线通过从潜在解决方案中剔除噪声数据，为未来自我偏好研究奠定基础。更广泛而言，本研究为不断增长的评估者偏差效应分类与隔离研究体系作出了贡献。