Learning to Optimize (L2O) has drawn increasing attention as it often remarkably accelerates the optimization procedure of complex tasks by ``overfitting" specific task type, leading to enhanced performance compared to analytical optimizers. Generally, L2O develops a parameterized optimization method (i.e., ``optimizer") by learning from solving sample problems. This data-driven procedure yields L2O that can efficiently solve problems similar to those seen in training, that is, drawn from the same ``task distribution". However, such learned optimizers often struggle when new test problems come with a substantially deviation from the training task distribution. This paper investigates a potential solution to this open challenge, by meta-training an L2O optimizer that can perform fast test-time self-adaptation to an out-of-distribution task, in only a few steps. We theoretically characterize the generalization of L2O, and further show that our proposed framework (termed as M-L2O) provably facilitates rapid task adaptation by locating well-adapted initial points for the optimizer weight. Empirical observations on several classic tasks like LASSO and Quadratic, demonstrate that M-L2O converges significantly faster than vanilla L2O with only $5$ steps of adaptation, echoing our theoretical results. Codes are available in https://github.com/VITA-Group/M-L2O.

翻译：摘要：学习优化（L2O）因其通过“过拟合”特定任务类型显著加速复杂任务优化过程而受到广泛关注，相较于解析优化器展现出更优性能。通常，L2O通过从求解样本问题中学习来开发参数化优化方法（即“优化器”）。这种数据驱动方法生成的L2O能够高效解决与训练中遇到的相似问题（即来自相同“任务分布”的数据）。然而，当新测试问题与训练任务分布存在显著偏差时，这类学习型优化器常常表现不佳。本文通过元训练一种L2O优化器来应对这一开放性挑战，该优化器能够在仅少数步骤内对分布外任务进行快速测试时自适应。我们从理论上刻画了L2O的泛化特性，并进一步证明我们提出的框架（称为M-L2O）能够通过定位优化器权重的良好适配初始点，显著促进快速任务自适应。在LASSO和二次型等经典任务上的实验观察表明，M-L2O在仅需5步自适应的情况下收敛速度显著快于传统L2O，这验证了我们的理论结果。代码可在https://github.com/VITA-Group/M-L2O获取。