Deep reinforcement learning (RL) works impressively in some environments and fails catastrophically in others. Ideally, RL theory should be able to provide an understanding of why this is, i.e. bounds predictive of practical performance. Unfortunately, current theory does not quite have this ability. We compare standard deep RL algorithms to prior sample complexity bounds by introducing a new dataset, BRIDGE. It consists of 155 deterministic MDPs from common deep RL benchmarks, along with their corresponding tabular representations, which enables us to exactly compute instance-dependent bounds. We choose to focus on deterministic environments because they share many interesting properties of stochastic environments, but are easier to analyze. Using BRIDGE, we find that prior bounds do not correlate well with when deep RL succeeds vs. fails, but discover a surprising property that does. When actions with the highest Q-values under the random policy also have the highest Q-values under the optimal policy (i.e. when it is optimal to be greedy on the random policy's Q function), deep RL tends to succeed; when they don't, deep RL tends to fail. We generalize this property into a new complexity measure of an MDP that we call the effective horizon, which roughly corresponds to how many steps of lookahead search would be needed in that MDP in order to identify the next optimal action, when leaf nodes are evaluated with random rollouts. Using BRIDGE, we show that the effective horizon-based bounds are more closely reflective of the empirical performance of PPO and DQN than prior sample complexity bounds across four metrics. We also find that, unlike existing bounds, the effective horizon can predict the effects of using reward shaping or a pre-trained exploration policy. Our code and data are available at https://github.com/cassidylaidlaw/effective-horizon

翻译：深度强化学习（RL）在某些环境中表现出色，而在其他环境中则彻底失败。理想情况下，RL理论应能理解这一现象的原因，即能够预测实际性能的界限。然而，当前理论尚不完全具备此能力。通过引入新数据集BRIDGE，我们将标准深度RL算法与先前的样本复杂度界限进行比较。该数据集包含来自常见深度RL基准测试的155个确定性马尔可夫决策过程（MDP）及其对应的表格表示，从而能够精确计算实例相关的界限。我们聚焦于确定性环境，因为它们与随机环境共享许多有趣性质，且更易分析。利用BRIDGE，我们发现先前的界限与深度RL成功或失败的相关性较差，而另一个令人惊讶的性质则具有相关性：若随机策略下具有最高Q值的动作在最优策略下也具有最高Q值（即对随机策略的Q函数采取贪心策略是最优的），则深度RL往往成功；反之则失败。我们将此性质推广为一种新的MDP复杂度度量，称为“有效视界”，其大致对应在此MDP中，若叶节点以随机展开评估，则需多少步前瞻搜索才能识别下一个最优动作。通过BRIDGE，我们证明基于有效视界的界限在四个指标上比先前的样本复杂度界限更能反映PPO和DQN的实际性能。我们还发现，与现有界限不同，有效视界能预测使用奖励塑造或预训练探索策略的效果。我们的代码和数据公开于https://github.com/cassidylaidlaw/effective-horizon。