In online reinforcement learning (online RL), balancing exploration and exploitation is crucial for finding an optimal policy in a sample-efficient way. To achieve this, existing sample-efficient online RL algorithms typically consist of three components: estimation, planning, and exploration. However, in order to cope with general function approximators, most of them involve impractical algorithmic components to incentivize exploration, such as optimization within data-dependent level-sets or complicated sampling procedures. To address this challenge, we propose an easy-to-implement RL framework called \textit{Maximize to Explore} (\texttt{MEX}), which only needs to optimize \emph{unconstrainedly} a single objective that integrates the estimation and planning components while balancing exploration and exploitation automatically. Theoretically, we prove that \texttt{MEX} achieves a sublinear regret with general function approximations for Markov decision processes (MDP) and is further extendable to two-player zero-sum Markov games (MG). Meanwhile, we adapt deep RL baselines to design practical versions of \texttt{MEX}, in both model-free and model-based manners, which can outperform baselines by a stable margin in various MuJoCo environments with sparse rewards. Compared with existing sample-efficient online RL algorithms with general function approximations, \texttt{MEX} achieves similar sample efficiency while enjoying a lower computational cost and is more compatible with modern deep RL methods.

翻译：在在线强化学习（在线RL）中，平衡探索与利用对于以样本高效的方式寻找最优策略至关重要。为实现这一目标，现有样本高效的在线RL算法通常包含三个组成部分：估计、规划与探索。然而，为应对通用函数逼近器，大多数算法需引入不切实际的算法组件来激励探索，例如在数据依赖的水平集内进行优化或采用复杂的采样程序。为解决这一挑战，我们提出一种易于实现的RL框架，称为《最大化以探索》（\texttt{MEX}），该框架仅需对融合估计与规划组件的单一目标进行无约束优化，同时自动平衡探索与利用。理论上，我们证明\texttt{MEX}在马尔可夫决策过程（MDP）中采用通用函数逼近时能够实现次线性遗憾，并且可进一步扩展到双人零和马尔可夫博弈（MG）。与此同时，我们基于深度RL基线方法设计了\texttt{MEX}的实用版本（包括无模型与基于模型两种形式），在多种具有稀疏奖励的MuJoCo环境中，该方法能稳定地超越基线方法。与现有采用通用函数逼近的样本高效在线RL算法相比，\texttt{MEX}在实现相似样本效率的同时，具有更低的计算成本，且与现代深度RL方法更兼容。