The temporal difference (TD) error was first formalized in Sutton (1988), where it was first characterized as the difference between temporally successive predictions, and later, in that same work, formulated as the difference between a bootstrapped target and a prediction. Since then, these two interpretations of the TD error have been used interchangeably in the literature, with the latter eventually being adopted as the standard critic loss in deep reinforcement learning (RL) architectures. In this work, we show that these two interpretations of the TD error are not always equivalent. In particular, we show that increasingly-nonlinear deep RL architectures can cause these interpretations of the TD error to yield increasingly different numerical values. Then, building on this insight, we show how choosing one interpretation of the TD error over the other can affect the performance of deep RL algorithms that utilize the TD error to compute other quantities, such as with deep differential (i.e., average-reward) RL methods. All in all, our results show that the default interpretation of the TD error as the difference between a bootstrapped target and a prediction does not always hold in deep RL settings.

翻译：时间差分（TD）误差最早由Sutton（1988）形式化定义，最初被描述为时间上连续预测之间的差值，随后在同一工作中被表述为自助法目标与预测之间的差值。此后，文献中对TD误差的这两种解释被交替使用，后者最终被采纳为深度强化学习（RL）架构中的标准评论者损失函数。在本研究中，我们证明TD误差的这两种解释并非总是等价的。具体而言，我们表明，非线性程度逐渐增强的深度RL架构会导致这两种TD误差解释产生数值差异的逐步增大。基于这一发现，我们进一步展示了在利用TD误差计算其他量（例如深度差分RL方法中的平均奖励）时，选择不同TD误差解释如何影响深度RL算法的性能。总之，我们的结果表明，将TD误差默认解释为自助法目标与预测之间的差值，在深度RL设定下并非始终成立。