In online reinforcement learning (online RL), balancing exploration and exploitation is crucial for finding an optimal policy in a sample-efficient way. To achieve this, existing sample-efficient online RL algorithms typically consist of three components: estimation, planning, and exploration. However, in order to cope with general function approximators, most of them involve impractical algorithmic components to incentivize exploration, such as optimization within data-dependent level-sets or complicated sampling procedures. To address this challenge, we propose an easy-to-implement RL framework called \textit{Maximize to Explore} (\texttt{MEX}), which only needs to optimize \emph{unconstrainedly} a single objective that integrates the estimation and planning components while balancing exploration and exploitation automatically. Theoretically, we prove that \texttt{MEX} achieves a sublinear regret with general function approximations for Markov decision processes (MDP) and is further extendable to two-player zero-sum Markov games (MG). Meanwhile, we adapt deep RL baselines to design practical versions of \texttt{MEX}, in both model-free and model-based manners, which can outperform baselines by a stable margin in various MuJoCo environments with sparse rewards. Compared with existing sample-efficient online RL algorithms with general function approximations, \texttt{MEX} achieves similar sample efficiency while enjoying a lower computational cost and is more compatible with modern deep RL methods.

翻译：在在线强化学习（online RL）中，平衡探索与利用对于以样本高效的方式找到最优策略至关重要。为实现这一目标，现有的样本高效在线强化学习算法通常包含三个组件：估计、规划和探索。然而，为应对通用函数逼近器，大多数算法需要引入不切实际的算法组件来激励探索，例如在数据依赖的水平集内进行优化或复杂的采样过程。为解决这一挑战，我们提出一个易于实现的强化学习框架，名为《最大化以探索》（\texttt{MEX}），该框架仅需优化一个 \emph{无约束} 的单一目标，该目标整合了估计与规划组件，同时自动平衡探索与利用。理论上，我们证明 \texttt{MEX} 在马尔可夫决策过程（MDP）中采用通用函数逼近时可实现次线性遗憾，并可进一步扩展到双人零和马尔可夫博弈（MG）。同时，我们基于深度强化学习基线，以无模型和基于模型两种方式设计了 \texttt{MEX} 的实用版本，在多种具有稀疏奖励的 MuJoCo 环境中，该框架能以稳定优势超越基线。与现有采用通用函数逼近的样本高效在线强化学习算法相比，\texttt{MEX} 在保持相似样本效率的同时，享有更低计算成本，并且与现代深度强化学习方法具有更高兼容性。