Stochastic optimal control, which has the goal of driving the behavior of noisy systems, is broadly applicable in science, engineering and artificial intelligence. Our work introduces Stochastic Optimal Control Matching (SOCM), a novel Iterative Diffusion Optimization (IDO) technique for stochastic optimal control that stems from the same philosophy as the conditional score matching loss for diffusion models. That is, the control is learned via a least squares problem by trying to fit a matching vector field. The training loss, which is closely connected to the cross-entropy loss, is optimized with respect to both the control function and a family of reparameterization matrices which appear in the matching vector field. The optimization with respect to the reparameterization matrices aims at minimizing the variance of the matching vector field. Experimentally, our algorithm achieves lower error than all the existing IDO techniques for stochastic optimal control for three out of four control problems, in some cases by an order of magnitude. The key idea underlying SOCM is the path-wise reparameterization trick, a novel technique that is of independent interest, e.g., for generative modeling. Code at https://github.com/facebookresearch/SOC-matching

翻译：随机最优控制旨在驱动含噪系统的行为，在科学、工程及人工智能领域具有广泛适用性。本文提出随机最优控制匹配（SOCM）——一种基于扩散模型条件评分匹配损失相同理念的新型迭代扩散优化（IDO）技术。该技术通过最小二乘问题学习最优控制，即拟合匹配向量场。其训练损失与交叉熵损失密切关联，同时针对控制函数及匹配向量场中的一族重参数化矩阵进行优化。对重参数化矩阵的优化旨在最小化匹配向量场的方差。实验表明，在四个控制问题中的三个问题上，本算法误差显著低于现有所有IDO技术，部分案例中误差降低一个数量级。SOCM的核心创新在于路径重参数化技巧——这一独立于本文的技术对生成建模等领域具有潜在价值。代码见https://github.com/facebookresearch/SOC-matching