Approaches to recommendation are typically evaluated in one of two ways: (1) via a (simulated) online experiment, often seen as the gold standard, or (2) via some offline evaluation procedure, where the goal is to approximate the outcome of an online experiment. Several offline evaluation metrics have been adopted in the literature, inspired by ranking metrics prevalent in the field of Information Retrieval. (Normalised) Discounted Cumulative Gain (nDCG) is one such metric that has seen widespread adoption in empirical studies, and higher (n)DCG values have been used to present new methods as the state-of-the-art in top-$n$ recommendation for many years. Our work takes a critical look at this approach, and investigates when we can expect such metrics to approximate the gold standard outcome of an online experiment. We formally present the assumptions that are necessary to consider DCG an unbiased estimator of online reward and provide a derivation for this metric from first principles, highlighting where we deviate from its traditional uses in IR. Importantly, we show that normalising the metric renders it inconsistent, in that even when DCG is unbiased, ranking competing methods by their normalised DCG can invert their relative order. Through a correlation analysis between off- and on-line experiments conducted on a large-scale recommendation platform, we show that our unbiased DCG estimates strongly correlate with online reward, even when some of the metric's inherent assumptions are violated. This statement no longer holds for its normalised variant, suggesting that nDCG's practical utility may be limited.

翻译：推荐方法的评估通常通过以下两种方式进行：(1) 通过（模拟）在线实验，这通常被视为黄金标准；或(2) 通过某种离线评估流程，其目标是近似在线实验的结果。文献中已采用多种受信息检索领域常用排序指标启发的离线评估指标。（归一化）折损累计增益（nDCG）便是其中一种在实证研究中被广泛采用的指标，多年来更高的（n）DCG值常被用于展示新方法在Top-$n$推荐领域达到最先进水平。本研究对此评估方式进行了批判性审视，并探讨了此类指标在何种条件下能够近似在线实验的黄金标准结果。我们形式化地提出了将DCG视为在线奖励无偏估计量所需的前提假设，并从第一性原理推导了该指标，重点指出了其与传统信息检索应用场景的差异。关键的是，我们证明了指标的归一化会导致其不一致性：即使DCG本身是无偏的，按归一化DCG对竞争方法排序也可能逆转它们的相对顺序。通过对大规模推荐平台进行的离线和在线实验的相关性分析，我们发现即使在该指标某些固有假设被违反的情况下，我们提出的无偏DCG估计仍与在线奖励强相关。这一结论对其归一化变体不再成立，表明nDCG的实际效用可能有限。