This work studies reinforcement learning (RL) in the context of multi-period supply chains subject to constraints, e.g., on production and inventory. We introduce Distributional Constrained Policy Optimization (DCPO), a novel approach for reliable constraint satisfaction in RL. Our approach is based on Constrained Policy Optimization (CPO), which is subject to approximation errors that in practice lead it to converge to infeasible policies. We address this issue by incorporating aspects of distributional RL into DCPO. Specifically, we represent the return and cost value functions using neural networks that output discrete distributions, and we reshape costs based on the associated confidence. Using a supply chain case study, we show that DCPO improves the rate at which the RL policy converges and ensures reliable constraint satisfaction by the end of training. The proposed method also improves predictability, greatly reducing the variance of returns between runs, respectively; this result is significant in the context of policy gradient methods, which intrinsically introduce significant variance during training.

翻译：本文研究在具有约束（如生产约束和库存约束）的多周期供应链中的强化学习（RL）。我们提出分布约束策略优化（DCPO），一种用于在强化学习中实现可靠约束满足的新方法。该方法基于约束策略优化（CPO），但CPO存在近似误差，在实践中可能导致其收敛到不可行的策略。我们通过将分布强化学习的要素纳入DCPO来解决此问题。具体而言，我们使用输出离散分布的神经网络表示回报和成本价值函数，并根据相应的置信度重塑成本。通过供应链案例研究，我们发现DCPO提升了强化学习策略的收敛速度，并确保训练结束时的可靠约束满足。该方法还改善了可预测性，大幅降低了不同运行之间回报的方差；在策略梯度方法（其本质上在训练过程中引入显著方差）的背景下，这一结果具有重要意义。