The rapid advancement of natural language processing, information retrieval (IR), computer vision, and other technologies has presented significant challenges in evaluating the performance of these systems. One of the main challenges is the scarcity of human-labeled data, which hinders the fair and accurate assessment of these systems. In this work, we specifically focus on evaluating IR systems with sparse labels, borrowing from recent research on evaluating computer vision tasks. taking inspiration from the success of using Fr\'echet Inception Distance (FID) in assessing text-to-image generation systems. We propose leveraging the Fr\'echet Distance to measure the distance between the distributions of relevant judged items and retrieved results. Our experimental results on MS MARCO V1 dataset and TREC Deep Learning Tracks query sets demonstrate the effectiveness of the Fr\'echet Distance as a metric for evaluating IR systems, particularly in settings where a few labels are available. This approach contributes to the advancement of evaluation methodologies in real-world scenarios such as the assessment of generative IR systems.

翻译：自然语言处理、信息检索、计算机视觉等技术的快速发展，给这些系统的性能评估带来了重大挑战。其中主要挑战之一是人工标注数据的稀缺性，这阻碍了系统公平准确的评估。本研究借鉴近期计算机视觉任务评估的研究成果，特别聚焦于稀疏标注场景下的信息检索系统评估。受Fréchet Inception Distance（FID）在文本生成图像系统评估中成功应用的启发，我们提出利用Fréchet距离测量相关判断项分布与检索结果分布之间的差异。在MS MARCO V1数据集及TREC深度学习轨道查询集上的实验结果表明，Fréchet距离作为评估信息检索系统的有效指标，尤其在仅有少量标注数据的情况下表现出色。该方法为生成式信息检索系统评估等现实场景中的评估方法论发展做出了贡献。