Efficient exploration is a central problem in reinforcement learning and is often formalized as maximizing the entropy of the state-action occupancy measure. While unconstrained maximum-entropy exploration is relatively well understood, real-world exploration is often constrained by safety, resource, or imitation requirements. This constrained setting is particularly challenging because entropy maximization lacks additive structure, rendering Bellman-equation-based methods inapplicable. Moreover, scalable approaches require policy parameterization, inducing non-convexity in both the objective and the constraints. To our knowledge, the only prior model-free policy-gradient approach for this setting under general policy parameterization is due to Ying et al. (2025). Unfortunately, their guarantees are limited to weak regret and ergodic averages, which do not imply that the final output is a single deployable policy that is near-optimal and nearly feasible. In this work we take a different approach to this problem, and propose Policy Gradient Penalty (PGP) method, a single-loop policy-space method that enforces general convex occupancy-measure constraints via quadratic-penalty regularization. PGP constructs pseudo-rewards that yield gradient estimates of the penalized objective, subsequently exploiting the classical Policy Gradient Theorem. We further establish the regularity of the penalized objective, providing the smoothness properties needed to justify the convergence of PGP. Leveraging hidden convexity and strong duality, we then establish global last-iterate convergence guarantees, attaining an $ε$-optimal constrained entropy value with $ε$ bounded constraint violation despite policy-induced non-convexity. We validate PGP through ablations on a grid-world benchmark and further demonstrate scalability on two challenging continuous-control tasks.

翻译：高效探索是强化学习中的核心问题，通常形式化为最大化状态-动作占有测度的熵。虽然无约束的最大熵探索相对易于理解，但实际中的探索往往受到安全性、资源或模仿要求的约束。这种有约束设置尤其具有挑战性，因为熵最大化缺乏可加结构，导致基于贝尔曼方程的方法无法适用。此外，可扩展的方法需要策略参数化，这会引入目标和约束中的非凸性。据我们所知，在一般策略参数化下，针对此设置的先前无模型策略梯度方法仅来自Ying等人（2025）。遗憾的是，他们的保证仅限于弱遗憾和遍历平均，这意味着最终输出并非一个接近最优且近乎可行的单一可部署策略。本研究采用不同方法解决该问题，提出策略梯度惩罚（PGP）方法，这是一种单循环策略空间方法，通过二次惩罚正则化来强制执行一般的凸占有测度约束。PGP构造伪奖励，从而获得惩罚目标的梯度估计，并随后利用经典策略梯度定理。我们进一步建立惩罚目标的正则性，提供证明PGP收敛所需的光滑性性质。利用隐藏凸性和强对偶性，我们随后证明了全局最终迭代收敛保证，在策略引起的非凸性下，达到带ε有界约束违反的ε-最优约束熵值。我们通过在网格世界基准上的消融实验验证PGP，并进一步在两个具有挑战性的连续控制任务中展示其可扩展性。