Large-scale AI evaluation increasingly relies on aggregating binary judgments from $K$ annotators, including LLMs used as judges. Most classical methods, e.g., Dawid-Skene or (weighted) majority voting, assume annotators are conditionally independent given the true label $Y\in\{0,1\}$, an assumption often violated by LLM judges due to shared data, architectures, prompts, and failure modes. Ignoring such dependencies can yield miscalibrated posteriors and even confidently incorrect predictions. We study label aggregation through a hierarchy of dependence-aware models based on Ising graphical models and latent factors. For class-dependent Ising models, the Bayes log-odds is generally quadratic in votes; for class-independent couplings, it reduces to a linear weighted vote with correlation-adjusted parameters. We present finite-$K$ examples showing that methods based on conditional independence can flip the Bayes label despite matching per-annotator marginals. We prove separation results demonstrating that these methods remain strictly suboptimal as the number of judges grows, incurring nonvanishing excess risk under latent factors. Finally, we evaluate the proposed method on three real-world datasets, demonstrating improved performance over the classical baselines.

翻译：大规模人工智能评估日益依赖于聚合来自$K$个标注者（包括用作评判者的大语言模型）的二元判断。大多数经典方法（例如Dawid-Skene模型或（加权）多数投票）假设标注者在给定真实标签$Y\in\{0,1\}$的条件下相互独立，这一假设常因大语言模型评判者共享数据、架构、提示词和失效模式而被违反。忽略此类依赖性可能导致后验概率校准失准，甚至产生自信的错误预测。我们通过基于伊辛图模型和潜在因子的层次化依赖感知模型研究标签聚合问题。对于类别依赖的伊辛模型，贝叶斯对数优势通常是投票数的二次函数；对于类别无关的耦合，则简化为具有相关性调整参数的线性加权投票。我们展示了有限$K$的实例，证明尽管满足每个标注者的边缘分布，基于条件独立性的方法仍可能翻转贝叶斯标签。我们证明了分离性结果，表明随着评判者数量增加，这些方法严格次优，在潜在因子下会产生非零的额外风险。最后，我们在三个真实数据集上评估所提方法，证明其性能优于经典基线。