Recommender systems are a ubiquitous feature of online platforms. Increasingly, they are explicitly tasked with increasing users' long-term satisfaction. In this context, we study a content exploration task, which we formalize as a multi-armed bandit problem with delayed rewards. We observe that there is an apparent trade-off in choosing the learning signal: Waiting for the full reward to become available might take several weeks, hurting the rate at which learning happens, whereas measuring short-term proxy rewards reflects the actual long-term goal only imperfectly. We address this challenge in two steps. First, we develop a predictive model of delayed rewards that incorporates all information obtained to date. Full observations as well as partial (short or medium-term) outcomes are combined through a Bayesian filter to obtain a probabilistic belief. Second, we devise a bandit algorithm that takes advantage of this new predictive model. The algorithm quickly learns to identify content aligned with long-term success by carefully balancing exploration and exploitation. We apply our approach to a podcast recommendation problem, where we seek to identify shows that users engage with repeatedly over two months. We empirically validate that our approach results in substantially better performance compared to approaches that either optimize for short-term proxies, or wait for the long-term outcome to be fully realized.

翻译：推荐系统是在线平台的一个普遍特征。它们越来越多地被明确要求提升用户的长期满意度。在此背景下，我们研究了一个内容探索任务，并将其形式化为一个具有延迟奖励的多臂赌博机问题。我们观察到，在选择学习信号时存在一个明显的权衡：等待完整奖励可能需要数周时间，从而损害学习的速度，而测量短期代理奖励则只能不完美地反映实际的长期目标。我们通过两个步骤应对这一挑战。首先，我们开发了一个延迟奖励的预测模型，该模型整合了迄今为止获得的所有信息。通过贝叶斯过滤器，将完整观测以及部分（短期或中期）结果相结合，以获得概率信念。其次，我们设计了一个利用这一新预测模型的赌博机算法。该算法通过谨慎平衡探索与利用，能够快速学习识别与长期成功相符的内容。我们将我们的方法应用于一个播客推荐问题，旨在识别用户能在两个月内反复参与的节目。我们通过实验验证，与优化短期代理或等待长期结果完全实现的方法相比，我们的方法能带来显著更好的性能。