The wide popularity of short videos on social media poses new opportunities and challenges to optimize recommender systems on the video-sharing platforms. Users sequentially interact with the system and provide complex and multi-faceted responses, including watch time and various types of interactions with multiple videos. One the one hand, the platforms aims at optimizing the users' cumulative watch time (main goal) in long term, which can be effectively optimized by Reinforcement Learning. On the other hand, the platforms also needs to satisfy the constraint of accommodating the responses of multiple user interactions (auxiliary goals) such like, follow, share etc. In this paper, we formulate the problem of short video recommendation as a Constrained Markov Decision Process (CMDP). We find that traditional constrained reinforcement learning algorithms can not work well in this setting. We propose a novel two-stage constrained actor-critic method: At stage one, we learn individual policies to optimize each auxiliary signal. At stage two, we learn a policy to (i) optimize the main signal and (ii) stay close to policies learned at the first stage, which effectively guarantees the performance of this main policy on the auxiliaries. Through extensive offline evaluations, we demonstrate effectiveness of our method over alternatives in both optimizing the main goal as well as balancing the others. We further show the advantage of our method in live experiments of short video recommendations, where it significantly outperforms other baselines in terms of both watch time and interactions. Our approach has been fully launched in the production system to optimize user experiences on the platform.

翻译：短视频在社交媒体上的广泛普及为优化视频分享平台推荐系统带来了新的机遇与挑战。用户与系统进行序列化交互，并产生复杂多方面的反馈，包括观看时长以及与多个视频之间的多种互动类型。一方面，平台致力于长期优化用户的累积观看时长（主要目标），强化学习能够有效实现这一优化。另一方面，平台还需满足多类用户交互行为反馈（辅助目标）的约束条件，例如关注、分享等。本文将短视频推荐问题建模为约束马尔可夫决策过程（Constrained Markov Decision Process, CMDP）。研究发现传统约束强化学习算法在此场景下表现不佳。我们提出一种新颖的两阶段约束Actor-Critic方法：第一阶段，学习针对各辅助信号的独立策略；第二阶段，学习一个策略，使其既能（i）优化主要信号，又能（ii）与第一阶段习得的策略保持接近，从而有效保证该主策略在辅助目标上的性能。通过大量离线评估，我们证明了该方法在优化主要目标及平衡其他目标方面均优于现有方法。我们进一步展示了该方法在短视频推荐在线实验中的优势：在观看时长和交互行为两个指标上均显著超越其他基线模型。目前该方案已在生产系统中全面部署，用于优化平台用户体验。