The theories of offline and online reinforcement learning, despite having evolved in parallel, have begun to show signs of the possibility for a unification, with algorithms and analysis techniques for one setting often having natural counterparts in the other. However, the notion of density ratio modeling, an emerging paradigm in offline RL, has been largely absent from online RL, perhaps for good reason: the very existence and boundedness of density ratios relies on access to an exploratory dataset with good coverage, but the core challenge in online RL is to collect such a dataset without having one to start. In this work we show -- perhaps surprisingly -- that density ratio-based algorithms have online counterparts. Assuming only the existence of an exploratory distribution with good coverage, a structural condition known as coverability (Xie et al., 2023), we give a new algorithm (GLOW) that uses density ratio realizability and value function realizability to perform sample-efficient online exploration. GLOW addresses unbounded density ratios via careful use of truncation, and combines this with optimism to guide exploration. GLOW is computationally inefficient; we complement it with a more efficient counterpart, HyGLOW, for the Hybrid RL setting (Song et al., 2022) wherein online RL is augmented with additional offline data. HyGLOW is derived as a special case of a more general meta-algorithm that provides a provable black-box reduction from hybrid RL to offline RL, which may be of independent interest.

翻译：离线与在线强化学习的理论虽平行发展，但已显现出统一的迹象——一种设置下的算法与分析技术往往自然对应另一种设置下的方法。然而，密度比建模（离线RL中的一个新兴范式）在在线RL中基本缺失，这或许有充分理由：密度比的存在性和有界性依赖于具有良好覆盖度的探索性数据集，而在线RL的核心挑战恰恰是在没有此类数据集的情况下收集它。本研究展示了——或许令人意外——基于密度比的算法存在在线对应版本。假设仅存在一个具有良好覆盖度的探索性分布（即称为“可覆盖性”的结构性条件，Xie等，2023），我们提出新算法GLOW，利用密度比可实现性与价值函数可实现性进行样本高效的在线探索。GLOW通过谨慎使用截断技术处理无界密度比，并结合乐观主义引导探索。该算法计算效率较低；我们为其补充了更高效的对应版本HyGLOW，用于混合RL场景（Song等，2022），其中在线RL由额外离线数据增强。HyGLOW可视为更通用元算法的特例，该元算法为从混合RL到离线RL提供了可证明的黑盒归约，可能具有独立的研究价值。