Recent advances in recommender systems have proved the potential of Reinforcement Learning (RL) to handle the dynamic evolution processes between users and recommender systems. However, learning to train an optimal RL agent is generally impractical with commonly sparse user feedback data in the context of recommender systems. To circumvent the lack of interaction of current RL-based recommender systems, we propose to learn a general Model-Agnostic Counterfactual Synthesis (MACS) Policy for counterfactual user interaction data augmentation. The counterfactual synthesis policy aims to synthesise counterfactual states while preserving significant information in the original state relevant to the user's interests, building upon two different training approaches we designed: learning with expert demonstrations and joint training. As a result, the synthesis of each counterfactual data is based on the current recommendation agent's interaction with the environment to adapt to users' dynamic interests. We integrate the proposed policy Deep Deterministic Policy Gradient (DDPG), Soft Actor Critic (SAC) and Twin Delayed DDPG in an adaptive pipeline with a recommendation agent that can generate counterfactual data to improve the performance of recommendation. The empirical results on both online simulation and offline datasets demonstrate the effectiveness and generalisation of our counterfactual synthesis policy and verify that it improves the performance of RL recommendation agents.

翻译：近期推荐系统的进展证明了强化学习在处理用户与推荐系统动态演化过程中的潜力。然而，在推荐系统中训练最优强化学习智能体通常因用户反馈数据稀疏而难以实现。为解决当前基于强化学习的推荐系统交互不足的问题，我们提出学习一种通用的模型无关反事实合成策略，用于反事实用户交互数据增强。该反事实合成策略旨在保留原始状态中与用户兴趣相关的重要信息的同时，合成反事实状态，并基于我们设计的两种训练方法——专家演示学习与联合训练。由此，每个反事实数据的合成均基于当前推荐智能体与环境的交互，以适应动态的用户兴趣。我们将所提出的策略与深度确定性策略梯度、软演员-评论家及双延迟深度确定性策略梯度集成至自适应流水线中，通过生成反事实数据提升推荐性能。在线模拟与离线数据集上的实验结果表明，我们的反事实合成策略具有良好的有效性与泛化性，并验证了其能提升强化学习推荐智能体的性能。