Incomplete relevance judgments limit the re-usability of test collections. When new systems are compared against previous systems used to build the pool of judged documents, they often do so at a disadvantage due to the ``holes'' in test collection (i.e., pockets of un-assessed documents returned by the new system). In this paper, we take initial steps towards extending existing test collections by employing Large Language Models (LLM) to fill the holes by leveraging and grounding the method using existing human judgments. We explore this problem in the context of Conversational Search using TREC iKAT, where information needs are highly dynamic and the responses (and, the results retrieved) are much more varied (leaving bigger holes). While previous work has shown that automatic judgments from LLMs result in highly correlated rankings, we find substantially lower correlates when human plus automatic judgments are used (regardless of LLM, one/two/few shot, or fine-tuned). We further find that, depending on the LLM employed, new runs will be highly favored (or penalized), and this effect is magnified proportionally to the size of the holes. Instead, one should generate the LLM annotations on the whole document pool to achieve more consistent rankings with human-generated labels. Future work is required to prompt engineering and fine-tuning LLMs to reflect and represent the human annotations, in order to ground and align the models, such that they are more fit for purpose.

翻译：不完整的相关性判断限制了测试集合的可重用性。当新系统与用于构建已判断文档池的先前系统进行比较时，往往会因测试集中的"空缺"（即新系统返回的未评估文档区域）处于劣势。本文迈出初步步骤，通过利用大型语言模型（LLM）并基于现有的人工判断方法，填补这些空缺以扩展现有测试集合。我们在对话搜索场景下（使用TREC iKAT数据集）探索该问题，此场景中信息需求高度动态，响应（及检索结果）更加多样化（从而造成更大空缺）。尽管先前研究表明LLM的自动判断能产生高度相关的排序，但我们发现当混合使用人工判断与自动判断时（无论采用单样本/少样本提示、微调还是不同LLM），相关性显著降低。进一步研究发现，根据所采用的LLM不同，新运行的检索结果会受到高度偏向（或惩罚），且这种效应随空缺规模成比例放大。相反，应生成整个文档池的LLM标注，以得到与人工标签更一致的排序。未来工作需要设计提示和微调LLM，使其反映并表征人工标注，从而对齐和校准模型，使其更符合实际应用需求。