Large Language Models (LLMs) have been used as relevance assessors for Information Retrieval (IR) evaluation collection creation due to reduced cost and increased scalability as compared to human assessors. While previous research has looked at the reliability of LLMs as compared to human assessors, in this work, we aim to understand if LLMs make systematic mistakes when judging relevance, rather than just understanding how good they are on average. To this aim, we propose a novel representational method for queries and documents that allows us to analyze relevance label distributions and compare LLM and human labels to identify patterns of disagreement and localize systematic areas of disagreement. We introduce a clustering-based framework that embeds query-document (Q-D) pairs into a joint semantic space, treating relevance as a relational property. Experiments on TREC Deep Learning 2019 and 2020 show that systematic disagreement between humans and LLMs is concentrated in specific semantic clusters rather than distributed randomly. Query-level analyses reveal recurring failures, most often in definition-seeking, policy-related, or ambiguous contexts. Queries with large variation in agreement across their clusters emerge as disagreement hotspots, where LLMs tend to under-recall relevant content or over-include irrelevant material. This framework links global diagnostics with localized clustering to uncover hidden weaknesses in LLM judgments, enabling bias-aware and more reliable IR evaluation.

翻译：大型语言模型（LLMs）因其相较于人工评估者具有更低的成本和更高的可扩展性，已被用作信息检索（IR）评估集合构建中的相关性评估工具。尽管先前的研究已关注LLMs相对于人工评估者的可靠性，但在本工作中，我们旨在探究LLMs在判断相关性时是否会出现系统性错误，而不仅仅是理解其平均表现水平。为此，我们提出了一种新颖的查询与文档表示方法，该方法使我们能够分析相关性标签分布，并通过比较LLM与人工标签来识别分歧模式并定位系统性的分歧区域。我们引入了一种基于聚类的框架，将查询-文档（Q-D）对嵌入到联合语义空间中，并将相关性视为一种关系属性。在TREC Deep Learning 2019和2020数据集上的实验表明，人类与LLMs之间的系统性分歧集中在特定的语义聚类中，而非随机分布。查询级别的分析揭示了重复出现的失败模式，最常见于定义寻求类、政策相关类或模糊语境下的查询。在其不同聚类中一致性差异较大的查询成为分歧热点，在这些情况下，LLMs倾向于对相关内容召回不足或过度包含无关材料。该框架通过将全局诊断与局部聚类相结合，揭示了LLM判断中的隐藏弱点，从而支持偏差感知且更可靠的IR评估。