A novel Policy Gradient (PG) algorithm, called Matryoshka Policy Gradient (MPG), is introduced and studied, in the context of max-entropy reinforcement learning, where an agent aims at maximising entropy bonuses additional to its cumulative rewards. MPG differs from standard PG in that it trains a sequence of policies to learn finite horizon tasks simultaneously, instead of a single policy for the single standard objective. For softmax policies, we prove convergence of MPG and global optimality of the limit by showing that the only critical point of the MPG objective is the optimal policy; these results hold true even in the case of continuous compact state space. MPG is intuitive, theoretically sound and we furthermore show that the optimal policy of the standard max-entropy objective can be approximated arbitrarily well by the optimal policy of the MPG framework. Finally, we justify that MPG is well suited when the policies are parametrized with neural networks and we provide an simple criterion to verify the global optimality of the policy at convergence. As a proof of concept, we evaluate numerically MPG on standard test benchmarks.

翻译：本文提出并研究了一种新颖的策略梯度（PG）算法，称为Matryoshka策略梯度（MPG），其应用于最大熵强化学习场景——智能体需要在累积奖励之外最大化熵增益。MPG与标准PG的区别在于，它训练一系列策略同时学习有限时域任务，而非针对单一标准目标训练单一策略。针对softmax策略，我们通过证明MPG目标的唯一临界点为最优策略，验证了MPG的收敛性及其极限的全局最优性；该结论在连续紧致状态空间下依然成立。MPG具备直观性、理论严谨性，且我们进一步表明，标准最大熵目标的最优策略可通过MPG框架的最优策略任意逼近。最后，我们论证了当策略采用神经网络参数化时MPG的良好适配性，并提供了一个简单准则用以验证收敛时策略的全局最优性。作为概念验证，我们在标准测试基准上对MPG进行了数值评估。