This paper studies reward-agnostic exploration in reinforcement learning (RL) -- a scenario where the learner is unware of the reward functions during the exploration stage -- and designs an algorithm that improves over the state of the art. More precisely, consider a finite-horizon non-stationary Markov decision process with $S$ states, $A$ actions, and horizon length $H$, and suppose that there are no more than a polynomial number of given reward functions of interest. By collecting an order of \begin{align*} \frac{SAH^3}{\varepsilon^2} \text{ sample episodes (up to log factor)} \end{align*} without guidance of the reward information, our algorithm is able to find $\varepsilon$-optimal policies for all these reward functions, provided that $\varepsilon$ is sufficiently small. This forms the first reward-agnostic exploration scheme in this context that achieves provable minimax optimality. Furthermore, once the sample size exceeds $\frac{S^2AH^3}{\varepsilon^2}$ episodes (up to log factor), our algorithm is able to yield $\varepsilon$ accuracy for arbitrarily many reward functions (even when they are adversarially designed), a task commonly dubbed as ``reward-free exploration.'' The novelty of our algorithm design draws on insights from offline RL: the exploration scheme attempts to maximize a critical reward-agnostic quantity that dictates the performance of offline RL, while the policy learning paradigm leverages ideas from sample-optimal offline RL paradigms.

翻译：本文研究强化学习中的奖赏无关探索场景——学习者在探索阶段不知晓奖赏函数——并设计了一种优于现有方法的算法。具体而言，考虑一个具有$S$个状态、$A$个动作、水平长度为$H$的有限水平非平稳马尔可夫决策过程，并假设存在不超过多项式多个给定的感兴趣奖赏函数。通过收集约 \begin{align*} \frac{SAH^3}{\varepsilon^2} \text{个样本片段（考虑对数因子）} \end{align*}，且在无奖赏信息指导的条件下，我们的算法能够为这些奖赏函数找到$\varepsilon$-最优策略，前提是$\varepsilon$足够小。这构成了该背景下首个实现可证明极小化最优性的奖赏无关探索方案。此外，一旦样本量超过 $\frac{S^2AH^3}{\varepsilon^2}$个片段（考虑对数因子），我们的算法能够为任意多个奖赏函数（即使是对抗性设计的）实现$\varepsilon$精度，这一任务通常被称为“奖赏无关探索”。我们算法设计的新颖性源于离线强化学习的洞察：探索方案旨在最大化一个关键的奖赏无关量，该量决定了离线强化学习的性能；而策略学习范式则借鉴了样本最优离线强化学习范式的思想。