Copious amounts of relevance judgments are necessary for the effective training and accurate evaluation of retrieval systems. Conventionally, these judgments are made by human assessors, rendering this process expensive and laborious. A recent study by Thomas et al. from Microsoft Bing suggested that large language models (LLMs) can accurately perform the relevance assessment task and provide human-quality judgments, but unfortunately their study did not yield any reusable software artifacts. Our work presents UMBRELA (a recursive acronym that stands for UMbrela is the Bing RELevance Assessor), an open-source toolkit that reproduces the results of Thomas et al. using OpenAI's GPT-4o model and adds more nuance to the original paper. Across Deep Learning Tracks from TREC 2019 to 2023, we find that LLM-derived relevance judgments correlate highly with rankings generated by effective multi-stage retrieval systems. Our toolkit is designed to be easily extensible and can be integrated into existing multi-stage retrieval and evaluation pipelines, offering researchers a valuable resource for studying retrieval evaluation methodologies. UMBRELA will be used in the TREC 2024 RAG Track to aid in relevance assessments, and we envision our toolkit becoming a foundation for further innovation in the field. UMBRELA is available at https://github.com/castorini/umbrela.

翻译：大量相关性判断对于检索系统的有效训练和准确评估至关重要。传统上，这些判断由人工评估者完成，使得该过程昂贵且费力。微软Bing的Thomas等人近期研究表明，大语言模型（LLM）能够准确执行相关性评估任务并提供人类水平的判断，但遗憾的是他们的研究未产生任何可复用的软件成果。本研究提出UMBRELA（一个递归缩写，代表UMbrela是Bing相关性评估器），这是一个开源工具包，它使用OpenAI的GPT-4o模型复现了Thomas等人的研究成果，并对原始论文进行了更细致的补充。在TREC 2019至2023的深度学习赛道中，我们发现LLM衍生的相关性判断与高效多阶段检索系统生成的排序结果高度相关。我们的工具包设计易于扩展，可集成到现有的多阶段检索与评估流程中，为研究者提供了研究检索评估方法学的宝贵资源。UMBRELA将用于TREC 2024 RAG赛道以辅助相关性评估，我们期望该工具包能成为该领域进一步创新的基础。UMBRELA可通过https://github.com/castorini/umbrela获取。