Following the introduction of Adam, several novel adaptive optimizers for deep learning have been proposed. These optimizers typically excel in some tasks but may not outperform Adam uniformly across all tasks. In this work, we introduce Meta-Adaptive Optimizers (MADA), a unified optimizer framework that can generalize several known optimizers and dynamically learn the most suitable one during training. The key idea in MADA is to parameterize the space of optimizers and dynamically search through it using hyper-gradient descent during training. We empirically compare MADA to other popular optimizers on vision and language tasks, and find that MADA consistently outperforms Adam and other popular optimizers, and is robust against sub-optimally tuned hyper-parameters. MADA achieves a greater validation performance improvement over Adam compared to other popular optimizers during GPT-2 training and fine-tuning. We also propose AVGrad, a modification of AMSGrad that replaces the maximum operator with averaging, which is more suitable for hyper-gradient optimization. Finally, we provide a convergence analysis to show that parameterized interpolations of optimizers can improve their error bounds (up to constants), hinting at an advantage for meta-optimizers.

翻译：自Adam优化器提出以来，深度学习领域已涌现出多种新颖的自适应优化器。这些优化器通常在特定任务上表现优异，但未必能在所有任务中一致地超越Adam。本文提出元自适应优化器（MADA），这是一个统一的优化器框架，能够泛化多种已知优化器，并在训练过程中动态学习最适合的优化器。MADA的核心思想是对优化器空间进行参数化，并在训练过程中通过超梯度下降进行动态搜索。我们在视觉和语言任务上实证比较了MADA与其他主流优化器，发现MADA始终优于Adam及其他流行优化器，并且对超参数调优不敏感。在GPT-2的训练与微调过程中，相较于其他流行优化器，MADA相比Adam带来了更大的验证性能提升。我们还提出了AVGrad——这是AMSGrad的一种改进版本，用平均算子替代了最大算子，使其更适用于超梯度优化。最后，我们通过收敛性分析表明，优化器的参数化插值能够改进其误差界（在常数因子范围内），这暗示了元优化器可能具有的理论优势。