Reinforcement learning (RL) theory has largely focused on proving minimax sample complexity bounds. These require strategic exploration algorithms that use relatively limited function classes for representing the policy or value function. Our goal is to explain why deep RL algorithms often perform well in practice, despite using random exploration and much more expressive function classes like neural networks. Our work arrives at an explanation by showing that many stochastic MDPs can be solved by performing only a few steps of value iteration on the random policy's Q function and then acting greedily. When this is true, we find that it is possible to separate the exploration and learning components of RL, making it much easier to analyze. We introduce a new RL algorithm, SQIRL, that iteratively learns a near-optimal policy by exploring randomly to collect rollouts and then performing a limited number of steps of fitted-Q iteration over those rollouts. Any regression algorithm that satisfies basic in-distribution generalization properties can be used in SQIRL to efficiently solve common MDPs. This can explain why deep RL works neural networks, since it is empirically established that neural networks generalize well in-distribution. Furthermore, SQIRL explains why random exploration works well in practice, since we show many environments can be solved by estimating the random policy's Q-function and then applying zero or a few steps of value iteration. We leverage SQIRL to derive instance-dependent sample complexity bounds for RL that are exponential only in an "effective horizon" of lookahead and on the complexity of the class used for function approximation. Empirically, we also find that SQIRL performance strongly correlates with PPO and DQN performance in a variety of stochastic environments, supporting that our theoretical analysis is predictive of practical performance.

翻译：强化学习理论主要关注极小极大样本复杂度界限的证明。这些理论要求采用策略性探索算法，且通常使用相对有限的函数类来表示策略或价值函数。我们的目标是解释为什么深度强化学习算法在实践中表现良好，尽管它们采用随机探索和更具表达能力（如神经网络）的函数类。我们的研究通过证明许多随机MDP只需对随机策略的Q函数执行少量几步值迭代然后采取贪心动作即可求解，从而得出解释。当这一条件成立时，我们发现可以将强化学习的探索和学习组件分离，从而极大简化分析过程。我们提出了一种新型强化学习算法SQIRL，该算法通过随机探索收集轨迹数据，并基于这些轨迹执行有限步数的拟合Q迭代，从而迭代学习近似最优策略。任何满足基本分布内泛化特性的回归算法均可用于SQIRL高效求解常见MDP。这可以解释深度强化学习在神经网络中的有效性——因为经验已证实神经网络在分布内具有良好的泛化能力。此外，SQIRL解释了随机探索在实践中表现良好的原因：我们证明许多环境只需估计随机策略的Q函数，再应用零步或几步值迭代即可求解。我们利用SQIRL推导了强化学习的实例相关样本复杂度界限，该复杂度仅与前瞻的"有效视界"和函数逼近所用函数类的复杂度成指数关系。实验表明，在多种随机环境中，SQIRL的性能与PPO和DQN具有强相关性，这支持了我们的理论分析对实际性能具有预测能力。