In recent years, there has been a growing interest in utilizing reinforcement learning (RL) to optimize long-term rewards in recommender systems. Since industrial recommender systems are typically designed as multi-stage systems, RL methods with a single agent face challenges when optimizing multiple stages simultaneously. The reason is that different stages have different observation spaces, and thus cannot be modeled by a single agent. To address this issue, we propose a novel UNidirectional-EXecution-based multi-agent Reinforcement Learning (UNEX-RL) framework to reinforce the long-term rewards in multi-stage recommender systems. We show that the unidirectional execution is a key feature of multi-stage recommender systems, bringing new challenges to the applications of multi-agent reinforcement learning (MARL), namely the observation dependency and the cascading effect. To tackle these challenges, we provide a cascading information chain (CIC) method to separate the independent observations from action-dependent observations and use CIC to train UNEX-RL effectively. We also discuss practical variance reduction techniques for UNEX-RL. Finally, we show the effectiveness of UNEX-RL on both public datasets and an online recommender system with over 100 million users. Specifically, UNEX-RL reveals a 0.558% increase in users' usage time compared with single-agent RL algorithms in online A/B experiments, highlighting the effectiveness of UNEX-RL in industrial recommender systems.

翻译：近年来，利用强化学习优化推荐系统长期奖励的研究日益受到关注。由于工业推荐系统通常被设计为多阶段系统，单智能体强化学习方法在同时优化多个阶段时面临挑战。这是因为不同阶段具有不同的观测空间，因此无法通过单一智能体进行建模。为解决这一问题，我们提出了一种基于单向执行的多智能体强化学习框架（UNEX-RL），旨在增强多阶段推荐系统的长期奖励。我们证明单向执行是多阶段推荐系统的关键特征，它为多智能体强化学习的应用带来了新挑战，即观测依赖性和级联效应。为应对这些挑战，我们提出级联信息链方法，将独立观测与动作依赖观测分离，并利用CIC有效训练UNEX-RL。我们还讨论了UNEX-RL的实用方差缩减技术。最后，我们在公开数据集和拥有超过1亿用户的在线推荐系统上验证了UNEX-RL的有效性。具体而言，在在线A/B实验中，UNEX-RL相比单智能体强化学习算法使用户使用时长提升了0.558%，突显了UNEX-RL在工业推荐系统中的有效性。