Stochastic optimal control, which has the goal of driving the behavior of noisy systems, is broadly applicable in science, engineering and artificial intelligence. Our work introduces Stochastic Optimal Control Matching (SOCM), a novel Iterative Diffusion Optimization (IDO) technique for stochastic optimal control that stems from the same philosophy as the conditional score matching loss for diffusion models. That is, the control is learned via a least squares problem by trying to fit a matching vector field. The training loss, which is closely connected to the cross-entropy loss, is optimized with respect to both the control function and a family of reparameterization matrices which appear in the matching vector field. The optimization with respect to the reparameterization matrices aims at minimizing the variance of the matching vector field. Experimentally, our algorithm achieves lower error than all the existing IDO techniques for stochastic optimal control for three out of four control problems, in some cases by an order of magnitude. The key idea underlying SOCM is the path-wise reparameterization trick, a novel technique that may be of independent interest. Code at https://github.com/facebookresearch/SOC-matching

翻译：随机最优控制旨在驱动噪声系统的行为，广泛应用于科学、工程和人工智能领域。我们的工作引入了随机最优控制匹配（SOCM），这是一种用于随机最优控制的新型迭代扩散优化（IDO）技术，其核心理念与扩散模型的条件得分匹配损失相同。即，通过尝试拟合匹配向量场，利用最小二乘问题来学习控制。该训练损失与交叉熵损失密切相关，并针对控制函数和匹配向量场中出现的一系列重参数化矩阵进行优化。针对重参数化矩阵的优化旨在最小化匹配向量场的方差。实验结果表明，在四个控制问题中的三个问题上，我们的算法实现的误差低于所有现有的用于随机最优控制的IDO技术，某些情况下误差降低了一个数量级。SOCM的关键思想是路径重参数化技巧，这一新技术可能具有独立的研究价值。代码地址：https://github.com/facebookresearch/SOC-matching