We propose a new approach to learned optimization where we represent the computation of an optimizer's update step using a neural network. The parameters of the optimizer are then learned by training on a set of optimization tasks with the objective to perform minimization efficiently. Our innovation is a new neural network architecture, Optimus, for the learned optimizer inspired by the classic BFGS algorithm. As in BFGS, we estimate a preconditioning matrix as a sum of rank-one updates but use a Transformer-based neural network to predict these updates jointly with the step length and direction. In contrast to several recent learned optimization-based approaches, our formulation allows for conditioning across the dimensions of the parameter space of the target problem while remaining applicable to optimization tasks of variable dimensionality without retraining. We demonstrate the advantages of our approach on a benchmark composed of objective functions traditionally used for the evaluation of optimization algorithms, as well as on the real world-task of physics-based visual reconstruction of articulated 3d human motion.

翻译：我们提出了一种新的学习优化方法，其中利用神经网络来表示优化器更新步骤的计算过程。优化器的参数随后通过在优化任务集上进行训练而学习，其目标是以高效方式执行最小化。我们的创新之处在于，受经典BFGS算法启发，为学习优化器设计了一种新型神经网络架构——Optimus。与BFGS类似，我们通过秩一更新之和来估计预处理矩阵，但采用基于Transformer的神经网络联合预测这些更新以及步长和方向。与近期几种基于学习优化的方法不同，我们的方案允许在目标问题参数空间的维度间进行条件调节，同时无需重新训练即可适用于可变维度的优化任务。我们在由传统优化算法评估目标函数构成的基准测试集以及基于物理的关节式三维人体运动视觉重建实际任务中，展示了本方法的优势。