State-of-the-art deep Q-learning methods update Q-values using state transition tuples sampled from the experience replay buffer. This strategy often uniformly and randomly samples or prioritizes data sampling based on measures such as the temporal difference (TD) error. Such sampling strategies can be inefficient at learning Q-function because a state's Q-value depends on the Q-value of successor states. If the data sampling strategy ignores the precision of the Q-value estimate of the next state, it can lead to useless and often incorrect updates to the Q-values. To mitigate this issue, we organize the agent's experience into a graph that explicitly tracks the dependency between Q-values of states. Each edge in the graph represents a transition between two states by executing a single action. We perform value backups via a breadth-first search starting from that expands vertices in the graph starting from the set of terminal states and successively moving backward. We empirically show that our method is substantially more data-efficient than several baselines on a diverse range of goal-reaching tasks. Notably, the proposed method also outperforms baselines that consume more batches of training experience and operates from high-dimensional observational data such as images.

翻译：最先进的深度Q学习方法通过从经验回放缓冲区中采样的状态转移元组来更新Q值。这种策略通常采用均匀随机采样，或基于时序差分（TD）误差等指标对数据采样进行优先级排序。此类采样策略在学习Q函数时可能效率低下，因为某一状态的Q值依赖于后续状态的Q值。如果数据采样策略忽略了下一状态Q值估计的精度，可能导致对Q值的无效甚至错误更新。为解决这一问题，我们将智能体的经验组织为一个显式跟踪状态间Q值依赖关系的图结构。图中每条边表示通过执行单个动作在两个状态之间进行的转移。我们通过从终止状态集合出发、逐步向后扩展图中顶点的广度优先搜索来执行值备份。实验表明，在多种目标到达任务中，我们的方法在数据效率上显著优于多个基线方法。值得注意的是，当使用图像等高维观测数据时，所提方法在消耗更多批量训练经验的基线方法中仍表现更优。