We derive a novel PAC-Bayesian generalization bound for reinforcement learning that explicitly accounts for Markov dependencies in the data, through the chain's mixing time. This contributes to overcoming challenges in obtaining generalization guarantees for reinforcement learning, where the sequential nature of data breaks the independence assumptions underlying classical bounds. The new bound provides non-vacuous certificates for modern off-policy algorithms such as Soft Actor-Critic. We demonstrate the practical utility of the bound through PB-SAC, a novel algorithm that optimizes the bound during training to guide exploration. Experiments across several continuous control tasks show that the proposed approach provides meaningful confidence certificates while maintaining competitive performance.

翻译：我们推导了一种新颖的强化学习PAC-Bayesian泛化界，该界通过马尔可夫链的混合时间显式地考虑了数据中的马尔可夫依赖性。这有助于克服为强化学习获取泛化保证的挑战，其中数据的序列性质打破了经典泛化界所依赖的独立性假设。该新界为现代离策略算法（如Soft Actor-Critic）提供了非平凡的保证证书。我们通过PB-SAC算法证明了该界的实际效用，PB-SAC是一种在训练过程中通过优化该界来指导探索的新算法。在多个连续控制任务上的实验表明，所提出的方法在保持竞争力的性能的同时，提供了有意义的置信度证书。