We propose a new approach to learned optimization where we represent the computation of an optimizer's update step using a neural network. The parameters of the optimizer are then learned by training on a set of optimization tasks with the objective to perform minimization efficiently. Our innovation is a new neural network architecture, Optimus, for the learned optimizer inspired by the classic BFGS algorithm. As in BFGS, we estimate a preconditioning matrix as a sum of rank-one updates but use a Transformer-based neural network to predict these updates jointly with the step length and direction. In contrast to several recent learned optimization-based approaches, our formulation allows for conditioning across the dimensions of the parameter space of the target problem while remaining applicable to optimization tasks of variable dimensionality without retraining. We demonstrate the advantages of our approach on a benchmark composed of objective functions traditionally used for the evaluation of optimization algorithms, as well as on the real world-task of physics-based visual reconstruction of articulated 3d human motion.

翻译：我们提出一种新的学习优化方法，其中优化器更新步骤的计算由神经网络表示。通过在求解一组优化任务时以高效最小化为目标进行训练，学习优化器的参数。我们的创新在于一种新型神经网络架构Optimus，其灵感源自经典BFGS算法，专为学习优化器设计。与BFGS类似，我们通过秩一更新之和估计预处理矩阵，但采用基于Transformer的神经网络联合预测这些更新及步长与方向。与近期多种基于学习优化的方法不同，我们的公式允许跨目标问题参数空间维度进行条件化，同时无需重新训练即可应用于可变维度的优化任务。我们在由传统优化算法评估目标函数构成的基准测试上，以及基于物理的铰接式三维人体运动视觉重建这一现实任务中，验证了该方法优势。