We study a class of reinforcement learning problems where the reward signals for policy learning are generated by an internal reward model that is dependent on and jointly optimized with the policy. This interdependence between the policy and the reward model leads to an unstable learning process because reward signals from an immature reward model are noisy and impede policy learning, and conversely, an under-optimized policy impedes reward estimation learning. We call this learning setting $\textit{Internally Rewarded Reinforcement Learning}$ (IRRL) as the reward is not provided directly by the environment but $\textit{internally}$ by a reward model. In this paper, we formally formulate IRRL and present a class of problems that belong to IRRL. We theoretically derive and empirically analyze the effect of the reward function in IRRL and based on these analyses propose the clipped linear reward function. Experimental results show that the proposed reward function can consistently stabilize the training process by reducing the impact of reward noise, which leads to faster convergence and higher performance compared with baselines in diverse tasks.

翻译：我们研究了一类强化学习问题，其中用于策略学习的奖励信号由一个内部奖励模型生成，该模型依赖于策略并与策略联合优化。策略与奖励模型之间的这种相互依赖导致学习过程不稳定，因为来自不成熟奖励模型的奖励信号带有噪声，会阻碍策略学习；反过来，优化不足的策略也会阻碍奖励估计学习。我们将这种学习设置称为"内部奖励强化学习"（IRRL），因为奖励并非直接由环境提供，而是由奖励模型"内部"生成。本文正式定义了IRRL，并介绍了一类属于IRRL的问题。我们从理论上推导并实证分析了IRRL中奖励函数的影响，基于这些分析提出了裁剪线性奖励函数。实验结果表明，所提出的奖励函数能够通过降低奖励噪声的影响来稳定训练过程，从而在多种任务中比基线方法实现更快的收敛速度和更高的性能。