In this paper, we introduce new formal methods and provide empirical evidence to highlight a unique safety concern prevalent in reinforcement learning (RL)-based recommendation algorithms -- 'user tampering.' User tampering is a situation where an RL-based recommender system may manipulate a media user's opinions through its suggestions as part of a policy to maximize long-term user engagement. We use formal techniques from causal modeling to critically analyze prevailing solutions proposed in the literature for implementing scalable RL-based recommendation systems, and we observe that these methods do not adequately prevent user tampering. Moreover, we evaluate existing mitigation strategies for reward tampering issues, and show that these methods are insufficient in addressing the distinct phenomenon of user tampering within the context of recommendations. We further reinforce our findings with a simulation study of an RL-based recommendation system focused on the dissemination of political content. Our study shows that a Q-learning algorithm consistently learns to exploit its opportunities to polarize simulated users with its early recommendations in order to have more consistent success with subsequent recommendations that align with this induced polarization. Our findings emphasize the necessity for developing safer RL-based recommendation systems and suggest that achieving such safety would require a fundamental shift in the design away from the approaches we have seen in the recent literature.

翻译：本文提出新的形式化方法，并通过实证证据揭示强化学习推荐算法中普遍存在的独特安全关切——"用户操控"。用户操控是指基于强化学习的推荐系统为最大化长期用户参与度，可能通过其推荐策略操控媒体用户观点的情况。我们采用因果建模的形式化技术，对现有文献中提出的可扩展强化学习推荐系统解决方案进行批判性分析，发现这些方法未能有效防范用户操控。进一步地，我们评估了现有针对奖励操控问题的缓解策略，证明这些方法在应对推荐场景下独特的用户操控现象时存在不足。我们通过聚焦政治内容传播的强化学习推荐系统仿真研究强化了上述发现。研究表明，Q学习算法会系统性利用早期推荐机会对模拟用户进行极化，从而确保后续与诱导极化方向一致的推荐获得更高成功率。我们的发现凸显了开发更安全的强化学习推荐系统的迫切性，并指出实现这种安全性需要从根本上改变近年来文献中常见的设计范式。