Serendipity in recommender systems (RSs) has attracted increasing attention as a concept that enhances user satisfaction by presenting unexpected and useful items. However, evaluating serendipitous performance remains challenging because its ground truth is generally unobservable. The existing offline metrics often depend on ambiguous definitions or are tailored to specific datasets and RSs, thereby limiting their generalizability. To address this issue, we propose a universally applicable evaluation framework that leverages large language models (LLMs) known for their extensive knowledge and reasoning capabilities, as evaluators. First, to improve the evaluation performance of the proposed framework, we assessed the serendipity prediction accuracy of LLMs using four different prompt strategies on a dataset containing user-annotated serendipitous ground truth and found that the chain-of-thought prompt achieved the highest accuracy. Next, we re-evaluated the serendipitous performance of both serendipity-oriented and general RSs using the proposed framework on three commonly used real-world datasets, without the ground truth. The results indicated that there was no serendipity-oriented RS that consistently outperformed across all datasets, and even a general RS sometimes achieved higher performance than the serendipity-oriented RS.

翻译：推荐系统中的意外性作为一种通过呈现意外且有用项目来提升用户满意度的概念，正受到越来越多的关注。然而，由于意外性的真实标注通常不可观测，评估其性能仍然具有挑战性。现有的离线指标往往依赖于模糊的定义，或是针对特定数据集和推荐系统定制，从而限制了其泛化能力。为解决这一问题，我们提出了一种普遍适用的评估框架，该框架利用具有广泛知识和推理能力的大型语言模型作为评估器。首先，为提升所提框架的评估性能，我们在一个包含用户标注的意外性真实数据的数据集上，使用四种不同的提示策略评估了LLMs的意外性预测准确性，发现思维链提示策略取得了最高的准确率。接着，我们在三个常用的真实世界数据集上，使用所提框架在没有真实标注的情况下，重新评估了面向意外性的推荐系统和通用推荐系统的意外性性能。结果表明，没有一种面向意外性的推荐系统能在所有数据集上持续优于其他系统，甚至通用推荐系统有时也能取得比面向意外性的推荐系统更高的性能。