Training reinforcement learning (RL) agents using scalar reward signals is often infeasible when an environment has sparse and non-Markovian rewards. Moreover, handcrafting these reward functions before training is prone to misspecification, especially when the environment's dynamics are only partially known. This paper proposes a novel pipeline for learning non-Markovian task specifications as succinct finite-state `task automata' from episodes of agent experience within unknown environments. We leverage two key algorithmic insights. First, we learn a product MDP, a model composed of the specification's automaton and the environment's MDP (both initially unknown), by treating the product MDP as a partially observable MDP and using the well-known Baum-Welch algorithm for learning hidden Markov models. Second, we propose a novel method for distilling the task automaton (assumed to be a deterministic finite automaton) from the learnt product MDP. Our learnt task automaton enables the decomposition of a task into its constituent sub-tasks, which improves the rate at which an RL agent can later synthesise an optimal policy. It also provides an interpretable encoding of high-level environmental and task features, so a human can readily verify that the agent has learnt coherent tasks with no misspecifications. In addition, we take steps towards ensuring that the learnt automaton is environment-agnostic, making it well-suited for use in transfer learning. Finally, we provide experimental results compared with two baselines to illustrate our algorithm's performance in different environments and tasks.

翻译：当环境具有稀疏且非马尔可夫奖励时，使用标量奖励信号训练强化学习（RL）代理通常不可行。此外，在训练前手工设计这些奖励函数容易导致规格错误，特别是当环境动态仅部分已知时。本文提出了一种新颖的流程，从未知环境中代理的经验片段中学习简洁的有限状态“任务自动机”，作为非马尔可夫任务规范。我们利用两个关键算法洞见。首先，通过将乘积MDP视为部分可观测MDP，并使用著名的Baum-Welch算法学习隐马尔可夫模型，我们学习一个乘积MDP，它是一个由规范自动机与环境MDP（两者最初均未知）组成的模型。其次，我们提出了一种从学习到的乘积MDP中提炼任务自动机（假设为确定性有限自动机）的新方法。我们学习到的任务自动机能够将任务分解为其构成子任务，从而提升RL代理后续合成最优策略的速度。它还提供了高层环境与任务特征的可解释编码，使人类能够轻松验证代理已学习到连贯且无规格错误的任务。此外，我们采取措施确保学习到的自动机与环境无关，使其非常适合迁移学习。最后，我们提供与两个基线相比的实验结果，以展示我们的算法在不同环境和任务中的性能。