Offline optimization paradigms such as offline Reinforcement Learning (RL) or Imitation Learning (IL) allow policy search algorithms to make use of offline data, but require careful incorporation of uncertainty in order to circumvent the challenges of distribution shift. Gradient-based policy search methods are a promising direction due to their effectiveness in high dimensions; however, we require a more careful consideration of how these methods interplay with uncertainty estimation. We claim that in order for an uncertainty metric to be amenable for gradient-based optimization, it must be (i) stably convergent to data when uncertainty is minimized with gradients, and (ii) not prone to underestimation of true uncertainty. We investigate smoothed distance to data as a metric, and show that it not only stably converges to data, but also allows us to analyze model bias with Lipschitz constants. Moreover, we establish an equivalence between smoothed distance to data and data likelihood, which allows us to use score-matching techniques to learn gradients of distance to data. Importantly, we show that offline model-based policy search problems that maximize data likelihood do not require values of likelihood; but rather only the gradient of the log likelihood (the score function). Using this insight, we propose Score-Guided Planning (SGP), a planning algorithm for offline RL that utilizes score-matching to enable first-order planning in high-dimensional problems, where zeroth-order methods were unable to scale, and ensembles were unable to overcome local minima. Website: https://sites.google.com/view/score-guided-planning/home

翻译：离线优化范式（如离线强化学习或模仿学习）允许策略搜索算法利用离线数据，但需谨慎融入不确定性以避免分布偏移带来的挑战。基于梯度的策略搜索方法因在高维空间中的有效性而前景广阔，但我们需要更细致地考量这些方法与不确定性估计的相互影响。我们认为，要使不确定性度量适用于梯度优化，它必须满足：(i) 在通过梯度最小化不确定性时能稳定收敛至数据；(ii) 不会低估真实不确定性。我们研究了到数据的平滑距离这一度量，并证明它不仅能稳定收敛至数据，还可通过Lipschitz常数分析模型偏差。此外，我们建立了到数据的平滑距离与数据似然之间的等价关系，从而可利用得分匹配技术学习到数据距离的梯度。重要的是，我们发现最大化数据似然的离线模型策略搜索问题无需似然的具体数值，而仅需对数似然的梯度（得分函数）。基于这一洞察，我们提出了得分引导规划（SGP）——一种面向离线强化学习的规划算法，它利用得分匹配在高维问题中实现一阶规划，而零阶方法在此类场景下难以扩展，集成方法也无法克服局部最小值。网站：https://sites.google.com/view/score-guided-planning/home