Training reinforcement learning (RL) policies from scratch is costly: it requires careful reward and environment design, extensive tuning, and substantial computation. Yet many control problems already have a functional but suboptimal policy available as a baseline. This paper proposes a method for embedding such a baseline into the RL training process, simultaneously improving training efficiency relative to from-scratch methods and producing a learning policy that outperforms the baseline. At each step, the method arbitrates between the baseline policy and a trainable learning policy, initially relying strongly on the baseline policy and then progressively transferring agency to the learning policy. By the end of training, the learning policy is a standalone neural network that operates without baseline policy support. The paper formalizes what it means for the baseline policy to be functional: under this policy, the agent reaches a goal set and remains there with high probability. The proposed arbitration mechanism is designed to exploit this property during training, yielding high goal-reaching rates right from the beginning of training. A theoretical analysis provides a formal interpretation of this behavior under stated assumptions and extends it to the final baseline-free regime, where explicit lower bounds are derived for the goal-reaching probability of the standalone learning policy. Empirical results on continuous-control benchmarks show that the proposed method achieves returns that match or exceed those of competitive approaches, while maintaining the highest goal-reaching rates throughout training among the compared methods -- including in the final stage, where the learning policy operates without any baseline support.

翻译：从头开始训练强化学习（RL）策略成本高昂：需要精细的奖励与环境设计、广泛的调参以及大量计算资源。然而，许多控制问题已存在一个功能完备但次优的基线策略。本文提出一种方法，将此类基线策略嵌入RL训练过程，既能提升训练效率（相较于从头训练的方法），又能产生优于基线策略的学习策略。该方法在每一步中，在基线策略与可训练的学习策略之间进行仲裁：初始阶段强依赖基线策略，随后逐步将智能体控制权转移给学习策略。训练结束时，学习策略将成为无需基线策略支持的独立神经网络。本文形式化定义了基线策略"功能完备"的含义：在该策略下，智能体能够以高概率抵达目标集并维持在该区域。所提出的仲裁机制旨在训练中利用此特性，使得从训练初期即能实现高目标到达率。理论分析在假设条件下为这一行为提供了形式化解释，并将其推广至最终的无基线阶段——在此阶段，推导出独立学习策略目标到达概率的显式下界。在连续控制基准上的实验结果表明：所提方法在回报上达到或超越竞争性方法，同时在整个训练过程中（包括最终无基线支持阶段）维持了最高目标到达率。