Training complex machine learning (ML) architectures requires a compute and time consuming process of selecting the right optimizer and tuning its hyper-parameters. A new paradigm of learning optimizers from data has emerged as a better alternative to hand-designed ML optimizers. We propose Mnemosyne optimizer, that uses Performers: implicit low-rank attention Transformers. It can learn to train entire neural network architectures including other Transformers without any task-specific optimizer tuning. We show that Mnemosyne: (a) generalizes better than popular LSTM optimizer, (b) in particular can successfully train Vision Transformers (ViTs) while meta--trained on standard MLPs and (c) can initialize optimizers for faster convergence in Robotics applications. We believe that these results open the possibility of using Transformers to build foundational optimization models that can address the challenges of regular Transformer training. We complement our results with an extensive theoretical analysis of the compact associative memory used by Mnemosyne.

翻译：训练复杂机器学习架构需要耗费大量算力和时间以选择合适的优化器并调整其超参数。一种从数据中学习优化器的新范式已涌现，成为手工设计机器学习优化器的更优替代方案。我们提出Memnosyne优化器，采用Performer（隐式低秩注意力Transformer）。该优化器无需针对特定任务调整优化器参数，即可学习训练包括其他Transformer在内的完整神经网络架构。研究表明Memnosyne：(a) 泛化能力优于流行的LSTM优化器；(b) 特别地，能够在基于标准MLP进行元训练后成功训练视觉Transformer（ViT）；(c) 可为机器人应用中的优化器初始化以实现更快的收敛速度。我们认为这些结果为利用Transformer构建基础优化模型开辟了可能性，可应对常规Transformer训练中的挑战。我们辅以对Memnosyne采用的紧凑关联记忆的广泛理论分析，对上述结果进行了补充论证。