Many real-world applications of reinforcement learning (RL) require the agent to deal with high-dimensional observations such as those generated from a megapixel camera. Prior work has addressed such problems with representation learning, through which the agent can provably extract endogenous, latent state information from raw observations and subsequently plan efficiently. However, such approaches can fail in the presence of temporally correlated noise in the observations, a phenomenon that is common in practice. We initiate the formal study of latent state discovery in the presence of such exogenous noise sources by proposing a new model, the Exogenous Block MDP (EX-BMDP), for rich observation RL. We start by establishing several negative results, by highlighting failure cases of prior representation learning based approaches. Then, we introduce the Predictive Path Elimination (PPE) algorithm, that learns a generalization of inverse dynamics and is provably sample and computationally efficient in EX-BMDPs when the endogenous state dynamics are near deterministic. The sample complexity of PPE depends polynomially on the size of the latent endogenous state space while not directly depending on the size of the observation space, nor the exogenous state space. We provide experiments on challenging exploration problems which show that our approach works empirically.

翻译：强化学习的许多实际应用(RL)要求代理处理高维的观测,例如从巨型像素相机产生的观测。先前的工作已经解决了代表性学习的这类问题,通过这种学习,该代理可以从原始观测中获取内在的、潜在的国家信息,并随后有效地进行规划。然而,在观测中出现与时间相关的噪音的情况下,这种方法可能会失败,这种现象在实践中是常见的现象。我们开始在这种外来噪音源的存在下,对潜伏状态发现进行正式研究,方法是提出一个新的模型,即外生性块MDP(EX-BMDP),用于丰富的观测。我们首先通过建立若干负面的结果,强调先前基于代表性学习方法的失败案例。然后,我们引入预测性路径消除算法,在发现反向动态时,当内生性状态的动态几乎具有威慑性时,在EX-BMDP中,这种方法具有可感知的抽样和计算效率。我们开始正式研究潜在内生空间的复杂度取决于潜在内生空间的大小,同时不直接取决于观测空间的大小,也不取决于外生空间的实验方法。我们提出了具有挑战性的探索性的问题。