Recommending novel content, which expands user horizons by introducing them to new interests, has been shown to improve users' long-term experience on recommendation platforms \cite{chen2021values}. Users however are not constantly looking to explore novel content. It is therefore crucial to understand their novelty-seeking intent and adjust the recommendation policy accordingly. Most existing literature models a user's propensity to choose novel content or to prefer a more diverse set of recommendations at individual interactions. Hierarchical structure, on the other hand, exists in a user's novelty-seeking intent, which is manifested as a static and intrinsic user preference for seeking novelty along with a dynamic session-based propensity. To this end, we propose a novel hierarchical reinforcement learning-based method to model the hierarchical user novelty-seeking intent, and to adapt the recommendation policy accordingly based on the extracted user novelty-seeking propensity. We further incorporate diversity and novelty-related measurement in the reward function of the hierarchical RL (HRL) agent to encourage user exploration \cite{chen2021values}. We demonstrate the benefits of explicitly modeling hierarchical user novelty-seeking intent in recommendations through extensive experiments on simulated and real-world datasets. In particular, we demonstrate that the effectiveness of our proposed hierarchical RL-based method lies in its ability to capture such hierarchically-structured intent. As a result, the proposed HRL model achieves superior performance on several public datasets, compared with state-of-art baselines.

翻译：推荐新颖内容能够扩展用户视野，引导其接触新兴趣领域，已被证明可提升用户在推荐平台上的长期体验\cite{chen2021values}。然而，用户并非始终寻求探索新内容。因此，理解用户的新颖性寻求意图并相应调整推荐策略至关重要。现有文献大多在单次交互层面建模用户选择新颖内容或偏好多样化推荐集的倾向。与此不同，用户的新颖性寻求意图存在层次化结构：静态的内在用户偏好与动态的会话级倾向共同构成其寻求新颖性的行为模式。为此，我们提出一种基于层次化强化学习的新方法，用于建模用户层次化新颖性寻求意图，并依据提取的用户新颖性寻求倾向自适应调整推荐策略。我们进一步将多样性与新颖性相关度量纳入层次化强化学习智能体的奖励函数中，以鼓励用户探索行为\cite{chen2021values}。通过在模拟数据集和真实世界数据集上的大量实验，我们证明了显式建模用户层次化新颖性寻求意图对推荐系统的提升效果。特别地，我们验证了所提层次化强化学习方法的有效性关键在于其捕获这种层次化结构意图的能力。由此，该方法在多个公开数据集上取得了优于现有基准模型的表现。