Recent advances in recommender systems have proved the potential of Reinforcement Learning (RL) to handle the dynamic evolution processes between users and recommender systems. However, learning to train an optimal RL agent is generally impractical with commonly sparse user feedback data in the context of recommender systems. To circumvent the lack of interaction of current RL-based recommender systems, we propose to learn a general Model-Agnostic Counterfactual Synthesis (MACS) Policy for counterfactual user interaction data augmentation. The counterfactual synthesis policy aims to synthesise counterfactual states while preserving significant information in the original state relevant to the user's interests, building upon two different training approaches we designed: learning with expert demonstrations and joint training. As a result, the synthesis of each counterfactual data is based on the current recommendation agent's interaction with the environment to adapt to users' dynamic interests. We integrate the proposed policy Deep Deterministic Policy Gradient (DDPG), Soft Actor Critic (SAC) and Twin Delayed DDPG in an adaptive pipeline with a recommendation agent that can generate counterfactual data to improve the performance of recommendation. The empirical results on both online simulation and offline datasets demonstrate the effectiveness and generalisation of our counterfactual synthesis policy and verify that it improves the performance of RL recommendation agents.

翻译：近期推荐系统的进展已证明强化学习在处理用户与推荐系统间动态演化过程中的潜力。然而，在推荐系统情境下，利用通常稀疏的用户反馈数据来训练最优强化学习智能体通常不切实际。为解决当前基于强化学习的推荐系统交互不足的问题，我们提出学习一种通用的模型无关反事实合成策略，用于反事实用户交互数据增强。该反事实合成策略旨在合成反事实状态，同时保留原始状态中与用户兴趣相关的重要信息，并基于我们设计的两种不同训练方法：基于专家示范的学习与联合训练。因此，每个反事实数据的合成均基于当前推荐智能体与环境的交互，以适应用户动态变化的兴趣。我们将所提出的策略与深度确定性策略梯度、软演员-评论家及双延迟深度确定性策略梯度集成为一个自适应流程，结合可生成反事实数据的推荐智能体，以提升推荐性能。在线仿真与离线数据集上的实证结果均证明了我们反事实合成策略的有效性与泛化能力，并验证其能提升强化学习推荐智能体的性能。