This paper studies stochastic control problems with the action space taken to be probability measures, with the objective penalised by the relative entropy. We identify suitable metric space on which we construct a gradient flow for the measure-valued control process, in the set of admissible controls, along which the cost functional is guaranteed to decrease. It is shown that any invariant measure of this gradient flow satisfies the Pontryagin optimality principle. If the problem we work with is sufficiently convex, the gradient flow converges exponentially fast. Furthermore, the optimal measure-valued control process admits a Bayesian interpretation which means that one can incorporate prior knowledge when solving such stochastic control problems. This work is motivated by a desire to extend the theoretical underpinning for the convergence of stochastic gradient type algorithms widely employed in the reinforcement learning community to solve control problems.

翻译：本文研究行动空间为概率测度且目标函数被相对熵惩罚的随机控制问题。我们确定了一个合适的度量空间，在该空间上针对可允许控制集内的测度值控制过程构建了一个梯度流，并确保代价泛函沿该梯度流下降。证明了该梯度流的任何不变测度均满足庞特里亚金最优性原理。当所研究问题具有充分凸性时，梯度流以指数速度收敛。此外，最优测度值控制过程具有贝叶斯解释，这意味着在求解此类随机控制问题时可以融入先验知识。本研究旨在为强化学习领域广泛使用的随机梯度类算法在求解控制问题时的收敛性提供理论支撑。