Natural gradient methods have been used to optimise the parameters of probability distributions in a variety of settings, often resulting in fast-converging procedures. Unfortunately, for many distributions of interest, computing the natural gradient has a number of challenges. In this work we propose a novel technique for tackling such issues, which involves reframing the optimisation as one with respect to the parameters of a surrogate distribution, for which computing the natural gradient is easy. We give several examples of existing methods that can be interpreted as applying this technique, and propose a new method for applying it to a wide variety of problems. Our method expands the set of distributions that can be efficiently targeted with natural gradients. Furthermore, it is fast, easy to understand, simple to implement using standard autodiff software, and does not require lengthy model-specific derivations. We demonstrate our method on maximum likelihood estimation and variational inference tasks.

翻译：自然梯度方法已被用于优化多种场景下概率分布的参数，通常能够实现快速收敛的过程。然而，对于许多感兴趣的分布而言，计算自然梯度面临诸多挑战。本文提出了一种解决此类问题的新技术，其核心思路是将优化问题重新表述为针对代理分布参数进行优化，而该代理分布的自然梯度容易计算。我们给出了若干现有方法作为该技术的实例，并提出了一种适用于广泛问题的新方法。该方法扩展了可通过自然梯度高效处理的分布范围，并且具有快速、易于理解、易于使用标准自动微分软件实现、无需针对模型进行冗长推导等优点。我们在最大似然估计和变分推断任务上验证了该方法的效果。