Dealing with unjudged documents ("holes") in relevance assessments is a perennial problem when evaluating search systems with offline experiments. Holes can reduce the apparent effectiveness of retrieval systems during evaluation and introduce biases in models trained with incomplete data. In this work, we explore whether large language models can help us fill such holes to improve offline evaluations. We examine an extreme, albeit common, evaluation setting wherein only a single known relevant document per query is available for evaluation. We then explore various approaches for predicting the relevance of unjudged documents with respect to a query and the known relevant document, including nearest neighbor, supervised, and prompting techniques. We find that although the predictions of these One-Shot Labelers (1SL) frequently disagree with human assessments, the labels they produce yield a far more reliable ranking of systems than the single labels do alone. Specifically, the strongest approaches can consistently reach system ranking correlations of over 0.86 with the full rankings over a variety of measures. Meanwhile, the approach substantially increases the reliability of t-tests due to filling holes in relevance assessments, giving researchers more confidence in results they find to be significant. Alongside this work, we release an easy-to-use software package to enable the use of 1SL for evaluation of other ad-hoc collections or systems.

翻译：在离线实验评估搜索系统时，相关性评估中存在的未判断文档（即"空洞"）是一个长期存在的问题。空洞可能降低检索系统在评估中的表面有效性，并在使用不完整数据训练的模型中引入偏差。本研究探索大型语言模型能否帮助填补此类空洞以改善离线评估。我们考察了一种极端但常见的评估场景：每个查询仅有一个已知相关文档可供评估。随后，我们探索了多种基于查询与已知相关文档来预测未判断文档相关性的方法，包括最近邻、监督学习和提示技术。研究发现，尽管这些一次性标注器（1SL）的预测结果常与人工评估相悖，但其所产生的标签相较于单一标签能更可靠地对系统进行排序。具体而言，最强方法在不同评估指标下均能实现与完整排序超过0.86的系统排序相关性。同时，该方法通过填补相关性评估中的空洞，显著提升了t检验的可靠性，使研究者对判定显著的结果更具信心。此外，我们随文发布了一套易用的软件包，以便将1SL应用于其他临时集合或系统的评估。