Reinforcement learning has been successful across several applications in which agents have to learn to act in environments with sparse feedback. However, despite this empirical success there is still a lack of theoretical understanding of how the parameters of reinforcement learning models and the features used to represent states interact to control the dynamics of learning. In this work, we use concepts from statistical physics, to study the typical case learning curves for temporal difference learning of a value function with linear function approximators. Our theory is derived under a Gaussian equivalence hypothesis where averages over the random trajectories are replaced with temporally correlated Gaussian feature averages and we validate our assumptions on small scale Markov Decision Processes. We find that the stochastic semi-gradient noise due to subsampling the space of possible episodes leads to significant plateaus in the value error, unlike in traditional gradient descent dynamics. We study how learning dynamics and plateaus depend on feature structure, learning rate, discount factor, and reward function. We then analyze how strategies like learning rate annealing and reward shaping can favorably alter learning dynamics and plateaus. To conclude, our work introduces new tools to open a new direction towards developing a theory of learning dynamics in reinforcement learning.

翻译：强化学习已在多个需要智能体在稀疏反馈环境中学习行动的领域取得成功。然而，尽管取得了实证成功，目前仍缺乏对强化学习模型参数与状态表征特征如何共同作用以控制学习动态的理论理解。本研究运用统计物理学概念，针对线性函数逼近器的价值函数时序差分学习，分析了典型情况下的学习曲线。我们的理论基于高斯等价假设，将随机轨迹的平均替换为时间相关的高斯特征平均，并在小规模马尔可夫决策过程中验证了假设。研究发现，与传统的梯度下降动态不同，由于对可能轨迹空间进行子采样而产生的随机半梯度噪声会导致价值误差出现显著平台期。我们分析了学习动态与平台期如何依赖特征结构、学习率、折扣因子及奖励函数，进而研究了学习率退火和奖励塑形等策略如何优化学习动态与平台期。本文的工作为发展强化学习动态理论提供了新工具和新方向。