Cranfield-style retrieval evaluations with too few or too many relevant documents or with low inter-assessor agreement on relevance can reduce the reliability of observations. In evaluations with human assessors, information needs are often formalized as retrieval topics to avoid an excessive number of relevant documents while maintaining good agreement. However, emerging evaluation setups that use Large Language Models (LLMs) as relevance assessors often use only queries, potentially decreasing the reliability. To study whether LLM relevance assessors benefit from formalized information needs, we synthetically formalize information needs with LLMs into topics that follow the established structure from previous human relevance assessments (i.e., descriptions and narratives). We compare assessors using synthetically formalized topics against the LLM-default query-only assessor on Robust04 and the 2019/2020 editions of TREC Deep Learning. We find that assessors without formalization judge many more documents relevant and have a lower agreement, leading to reduced reliability in retrieval evaluations. Furthermore, we show that the formalized topics improve agreement between human and LLM relevance judgments, even when the topics are not highly similar to their human counterparts. Our findings indicate that LLM relevance assessors should use formalized information needs, as is standard for human assessment, and synthetically formalize topics when no human formalization exists to improve evaluation reliability.

翻译：克兰菲尔德式检索评估若存在相关文档过少或过多，或评估者间相关性判断一致性较低的问题，将降低观察结果的可靠性。在人工评估中，信息需求常被形式化为检索主题，以避免过多相关文档并保持良好一致性。然而，新兴采用大语言模型作为相关性评估者的评估设置常仅使用查询，可能降低可靠性。为探究大语言模型相关性评估者是否受益于形式化信息需求，我们利用大语言模型将信息需求综合形式化为遵循先前人工相关性评估既定结构（即描述与叙述）的主题。在Robust04及2019/2020版TREC深度学习数据集上，我们对比了使用综合形式化主题的评估者与默认仅使用查询的大语言模型评估者。研究发现，未形式化主题的评估者判定更多文档为相关且一致性较低，导致检索评估可靠性下降。此外，我们证明即使形式化主题与人工主题高度不相似，也能提升人与大语言模型之间相关性判断的一致性。研究结果表明，大语言模型相关性评估者应采用形式化信息需求（如人工评估标准流程），并在无人为形式化时综合生成主题，以提高评估可靠性。