Reinforcement learning policies are typically represented by black-box neural networks, which are non-interpretable and not well-suited for safety-critical domains. To address both of these issues, we propose constrained normalizing flow policies as interpretable and safe-by-construction policy models. We achieve safety for reinforcement learning problems with instantaneous safety constraints, for which we can exploit domain knowledge by analytically constructing a normalizing flow that ensures constraint satisfaction. The normalizing flow corresponds to an interpretable sequence of transformations on action samples, each ensuring alignment with respect to a particular constraint. Our experiments reveal benefits beyond interpretability in an easier learning objective and maintained constraint satisfaction throughout the entire learning process. Our approach leverages constraints over reward engineering while offering enhanced interpretability, safety, and direct means of providing domain knowledge to the agent without relying on complex reward functions.

翻译：强化学习策略通常由黑箱神经网络表示，这些网络缺乏可解释性，且不适用于安全关键领域。为解决这两个问题，我们提出将约束归一化流策略作为具备可解释性与内在安全性的策略模型。针对具有瞬时安全约束的强化学习问题，我们通过解析构建归一化流实现约束满足，从而可利用领域知识。该归一化流对应于对动作样本施加的可解释变换序列，每个变换均确保与特定约束对齐。实验表明，该方法不仅提升可解释性，还能简化学习目标并保证整个学习过程中约束始终满足。我们的方法在增强可解释性与安全性的同时，通过奖励工程实现约束，并提供无需复杂奖励函数即可向智能体注入领域知识的直接途径。