Since Rendle and Krichene argued that commonly used sampling-based evaluation metrics are "inconsistent" with respect to the global metrics (even in expectation), there have been a few studies on the sampling-based recommender system evaluation. Existing methods try either mapping the sampling-based metrics to their global counterparts or more generally, learning the empirical rank distribution to estimate the top-$K$ metrics. However, despite existing efforts, there is still a lack of rigorous theoretical understanding of the proposed metric estimators, and the basic item sampling also suffers from the "blind spot" issue, i.e., estimation accuracy to recover the top-$K$ metrics when $K$ is small can still be rather substantial. In this paper, we provide an in-depth investigation into these problems and make two innovative contributions. First, we propose a new item-sampling estimator that explicitly optimizes the error with respect to the ground truth, and theoretically highlight its subtle difference against prior work. Second, we propose a new adaptive sampling method which aims to deal with the "blind spot" problem and also demonstrate the expectation-maximization (EM) algorithm can be generalized for such a setting. Our experimental results confirm our statistical analysis and the superiority of the proposed works. This study helps lay the theoretical foundation for adopting item sampling metrics for recommendation evaluation, and provides strong evidence towards making item sampling a powerful and reliable tool for recommendation evaluation.

翻译：自Rendle和Krichene指出常用的基于采样的评估指标相对于全局指标（即使在期望意义上）具有"不一致性"以来，已出现少量关于基于采样的推荐系统评估研究。现有方法要么试图将基于采样的指标映射到其全局对应指标，要么更一般地通过学习经验排序分布来估计前K个指标。然而，尽管已有诸多努力，但针对所提出的指标估计器仍缺乏严格的理论理解，且基础物品采样方法存在"盲点"问题——即当K值较小时，恢复前K个指标的估计精度仍可能相当不足。本文深入探究了这些问题并做出两项创新贡献。首先，我们提出一种新的物品采样估计器，该估计器显式优化了相对于真实值的误差，并从理论上阐明了其与先前工作的细微差异。其次，我们提出一种新的自适应采样方法，旨在解决"盲点"问题，并证明期望最大化算法可推广至此类场景。实验结果验证了我们的统计分析及所提方法的优越性。本研究为采用物品采样指标进行推荐评估奠定了理论基础，并为将物品采样打造为推荐评估中强大且可靠的工具提供了有力证据。