The evaluation of recommendation systems is a complex task. The offline and online evaluation metrics for recommender systems are ambiguous in their true objectives. The majority of recently published papers benchmark their methods using ill-posed offline evaluation methodology that often fails to predict true online performance. Because of this, the impact that academic research has on the industry is reduced. The aim of our research is to investigate and compare the online performance of offline evaluation metrics. We show that penalizing popular items and considering the time of transactions during the evaluation significantly improves our ability to choose the best recommendation model for a live recommender system. Our results, averaged over five large-size real-world live data procured from recommenders, aim to help the academic community to understand better offline evaluation and optimization criteria that are more relevant for real applications of recommender systems.

翻译：推荐系统的评估是一项复杂的任务。离线与在线评估指标在真实目标上存在模糊性。近期发表的大多数论文采用不适定的离线评估方法对其方法进行基准测试，这往往无法预测真实的在线性能。因此，学术研究对工业界的影响力有所降低。本研究旨在探究并比较离线评估指标的在线性能。我们证明，在评估过程中对流行物品进行惩罚并考虑交易时间，能显著提升我们为实时推荐系统选择最优推荐模型的能力。基于从推荐系统获取的五个大规模真实在线数据的平均结果，本研究旨在帮助学术界更好地理解与推荐系统实际应用更相关的离线评估和优化标准。