Reinforcement learning-based recommender systems have recently gained popularity. However, the design of the reward function, on which the agent relies to optimize its recommendation policy, is often not straightforward. Exploring the causality underlying users' behavior can take the place of the reward function in guiding the agent to capture the dynamic interests of users. Moreover, due to the typical limitations of simulation environments (e.g., data inefficiency), most of the work cannot be broadly applied in large-scale situations. Although some works attempt to convert the offline dataset into a simulator, data inefficiency makes the learning process even slower. Because of the nature of reinforcement learning (i.e., learning by interaction), it cannot collect enough data to train during a single interaction. Furthermore, traditional reinforcement learning algorithms do not have a solid capability like supervised learning methods to learn from offline datasets directly. In this paper, we propose a new model named the causal decision transformer for recommender systems (CDT4Rec). CDT4Rec is an offline reinforcement learning system that can learn from a dataset rather than from online interaction. Moreover, CDT4Rec employs the transformer architecture, which is capable of processing large offline datasets and capturing both short-term and long-term dependencies within the data to estimate the causal relationship between action, state, and reward. To demonstrate the feasibility and superiority of our model, we have conducted experiments on six real-world offline datasets and one online simulator.

翻译：基于强化学习的推荐系统近年来受到广泛关注。然而，智能体用于优化推荐策略的奖励函数设计往往并非易事。探索用户行为背后的因果性可替代奖励函数，引导智能体捕捉用户的动态兴趣。此外，受限于仿真环境（如数据效率低下）的典型缺陷，多数工作难以在大规模场景中广泛应用。尽管部分研究尝试将离线数据集转化为模拟器，但数据低效性反而使学习进程更加缓慢。由于强化学习的本质（即通过交互进行学习），使其无法在单次交互过程中收集足够训练数据。更关键的是，传统强化学习算法不具备监督学习方法那样直接学习离线数据集的稳健能力。本文提出名为面向推荐系统的因果决策Transformer（CDT4Rec）的新模型。CDT4Rec是一种离线强化学习系统，能够从数据集中而非在线交互中学习。同时，CDT4Rec采用Transformer架构，可处理大规模离线数据集并捕获数据中的短期与长期依赖关系，以估计动作、状态与奖励之间的因果关系。为验证模型可行性与优越性，我们在六个真实世界离线数据集和一个在线模拟器上开展了实验。