This paper investigates conservative exploration in reinforcement learning where the performance of the learning agent is guaranteed to be above a certain threshold throughout the learning process. It focuses on the tabular episodic Markov Decision Process (MDP) setting that has finite states and actions. With the knowledge of an existing safe baseline policy, an algorithm termed as StepMix is proposed to balance the exploitation and exploration while ensuring that the conservative constraint is never violated in each episode with high probability. StepMix features a unique design of a mixture policy that adaptively and smoothly interpolates between the baseline policy and the optimistic policy. Theoretical analysis shows that StepMix achieves near-optimal regret order as in the constraint-free setting, indicating that obeying the stringent episode-wise conservative constraint does not compromise the learning performance. Besides, a randomization-based EpsMix algorithm is also proposed and shown to achieve the same performance as StepMix. The algorithm design and theoretical analysis are further extended to the setting where the baseline policy is not given a priori but must be learned from an offline dataset, and it is proved that similar conservative guarantee and regret can be achieved if the offline dataset is sufficiently large. Experiment results corroborate the theoretical analysis and demonstrate the effectiveness of the proposed conservative exploration strategies.

翻译：本文研究了强化学习中的保守探索问题，要求学习智能体在整个学习过程中的性能始终不低于特定阈值。研究聚焦于具有有限状态和动作的表格化回合制马尔可夫决策过程（MDP）场景。基于现有安全基线策略的知识，本文提出了一种名为StepMix的算法，该算法在确保高概率下每回合保守约束不被违反的同时，平衡了利用与探索。StepMix的独特之处在于设计了混合策略，该策略能够自适应且平滑地在基线策略与乐观策略之间进行插值。理论分析表明，StepMix在无约束场景下实现了近优的遗憾阶数，表明遵守严格的回合制保守约束不会损害学习性能。此外，本文还提出了一种基于随机化的EpsMix算法，并证明其能达到与StepMix相同的性能。算法设计与理论分析进一步扩展至基线策略并非先验已知而需从离线数据集中学习的情形，并证明了若离线数据集足够大，则可实现类似的保守保证与遗憾。实验结果佐证了理论分析，并验证了所提保守探索策略的有效性。