Safe reinforcement learning (RL) with hard constraint guarantees is a promising optimal control direction for multi-energy management systems. It only requires the environment-specific constraint functions itself a prior and not a complete model (i.e. plant, disturbance and noise models, and prediction models for states not included in the plant model - e.g. demand, weather, and price forecasts). The project-specific upfront and ongoing engineering efforts are therefore still reduced, better representations of the underlying system dynamics can still be learned and modeling bias is kept to a minimum (no model-based objective function). However, even the constraint functions alone are not always trivial to accurately provide in advance (e.g. an energy balance constraint requires the detailed determination of all energy inputs and outputs), leading to potentially unsafe behavior. In this paper, we present two novel advancements: (I) combining the Optlayer and SafeFallback method, named OptLayerPolicy, to increase the initial utility while keeping a high sample efficiency. (II) introducing self-improving hard constraints, to increase the accuracy of the constraint functions as more data becomes available so that better policies can be learned. Both advancements keep the constraint formulation decoupled from the RL formulation, so that new (presumably better) RL algorithms can act as drop-in replacements. We have shown that, in a simulated multi-energy system case study, the initial utility is increased to 92.4% (OptLayerPolicy) compared to 86.1% (OptLayer) and that the policy after training is increased to 104.9% (GreyOptLayerPolicy) compared to 103.4% (OptLayer) - all relative to a vanilla RL benchmark. While introducing surrogate functions into the optimization problem requires special attention, we do conclude that the newly presented GreyOptLayerPolicy method is the most advantageous.

翻译：具有硬约束保障的安全强化学习是多能源管理系统一个有前景的最优控制方向。该方法仅需预先知道环境特定的约束函数本身，而无需完整模型（即对象模型、扰动和噪声模型，以及未包含在对象模型中的状态预测模型——例如需求、天气和价格预测）。因此，项目特定的前期和持续工程工作量得以降低，底层系统动力学的更优表示仍可被学习，且建模偏差保持最小（无基于模型的目标函数）。然而，即使仅约束函数本身也并非总能轻易准确提前提供（例如能量平衡约束需要详细确定所有能量输入和输出），这可能导致不安全行为。本文提出两项新进展：（I）将OptLayer方法与SafeFallback方法相结合，命名为OptLayerPolicy，以提高初始效用并保持高样本效率；（II）引入自改进硬约束，随着更多数据可用而提高约束函数的准确性，从而学习更优策略。这两项进展均保持约束公式与强化学习公式解耦，使得新的（可能更优的）强化学习算法可作为即插即用替代品。我们在模拟多能源系统案例研究中表明，初始效用提升至92.4%（OptLayerPolicy），而OptLayer为86.1%；训练后策略提升至104.9%（GreyOptLayerPolicy），而OptLayer为103.4%——所有结果均以朴素强化学习基准为参照。尽管在优化问题中引入替代函数需要特别关注，但我们得出结论：新提出的GreyOptLayerPolicy方法最为有利。