We propose a refinement of temporal-difference learning that enforces first-order Bellman consistency: the learned value function is trained to match not only the Bellman targets in value but also their derivatives with respect to states and actions. By differentiating the Bellman backup through differentiable dynamics, we obtain analytically consistent gradient targets. Incorporating these into the critic objective using a Sobolev-type loss encourages the critic to align with both the value and local geometry of the target function. This first-order TD matching principle can be seamlessly integrated into existing algorithms, such as Q-learning or actor-critic methods (e.g., DDPG, SAC), potentially leading to faster critic convergence and more stable policy gradients without altering their overall structure.

翻译：我们提出了一种时间差分学习的改进方法，该方法强制实现一阶贝尔曼一致性：学习的价值函数不仅被训练以匹配贝尔曼目标的价值，还需匹配其关于状态和动作的导数。通过对可微分动态系统的贝尔曼备份进行微分，我们获得了解析一致性的梯度目标。将这些目标通过Sobolev型损失纳入评论家目标函数中，促使评论家与目标函数的价值及局部几何结构对齐。这一阶时间差分匹配原则可无缝集成到现有算法中，例如Q学习或演员-评论家方法（如DDPG、SAC），在不改变其整体结构的前提下，可能实现更快的评论家收敛和更稳定的策略梯度。