Reinforcement learning has been successful across several applications in which agents have to learn to act in environments with sparse feedback. However, despite this empirical success there is still a lack of theoretical understanding of how the parameters of reinforcement learning models and the features used to represent states interact to control the dynamics of learning. In this work, we use concepts from statistical physics, to study the typical case learning curves for temporal difference learning of a value function with linear function approximators. Our theory is derived under a Gaussian equivalence hypothesis where averages over the random trajectories are replaced with temporally correlated Gaussian feature averages and we validate our assumptions on small scale Markov Decision Processes. We find that the stochastic semi-gradient noise due to subsampling the space of possible episodes leads to significant plateaus in the value error, unlike in traditional gradient descent dynamics. We study how learning dynamics and plateaus depend on feature structure, learning rate, discount factor, and reward function. We then analyze how strategies like learning rate annealing and reward shaping can favorably alter learning dynamics and plateaus. To conclude, our work introduces new tools to open a new direction towards developing a theory of learning dynamics in reinforcement learning.

翻译：强化学习在多个需要智能体在稀疏反馈环境中学习行动的应用中取得了成功。然而，尽管有这些经验上的成功，目前仍缺乏对强化学习模型参数与用于表示状态的特征如何相互作用以控制学习动力学的理论理解。在本工作中，我们利用统计物理的概念，研究了使用线性函数逼近器进行值函数时间差分学习的典型案例学习曲线。我们的理论在高斯等价性假设下推导，其中随机轨迹上的平均值被替换为时间相关的相关高斯特征平均值，并在小规模马尔可夫决策过程上验证了我们的假设。我们发现，由于对可能轨迹空间进行子采样而产生的随机半梯度噪声会导致值误差出现显著的平台期，这与传统梯度下降动力学不同。我们研究了学习动力学和平台期如何依赖于特征结构、学习率、折扣因子和奖励函数。接着，我们分析了学习率退火和奖励塑造等策略如何有利地改变学习动力学和平台期。总之，我们的工作引入了新工具，为发展强化学习中学习动力学的理论开辟了新方向。