Lion (Evolved Sign Momentum), a new optimizer discovered through program search, has shown promising results in training large AI models. It performs comparably or favorably to AdamW but with greater memory efficiency. As we can expect from the results of a random search program, Lion incorporates elements from several existing algorithms, including signed momentum, decoupled weight decay, Polak, and Nesterov momentum, but does not fit into any existing category of theoretically grounded optimizers. Thus, even though Lion appears to perform well as a general-purpose optimizer for a wide range of tasks, its theoretical basis remains uncertain. This lack of theoretical clarity limits opportunities to further enhance and expand Lion's efficacy. This work aims to demystify Lion. Based on both continuous-time and discrete-time analysis, we demonstrate that Lion is a theoretically novel and principled approach for minimizing a general loss function $f(x)$ while enforcing a bound constraint $\|x\|_\infty \leq 1/\lambda$. Lion achieves this through the incorporation of decoupled weight decay, where $\lambda$ represents the weight decay coefficient. Our analysis is made possible by the development of a new Lyapunov function for the Lion updates. It applies to a broader family of Lion-$\kappa$ algorithms, where the $\text{sign}(\cdot)$ operator in Lion is replaced by the subgradient of a convex function $\kappa$, leading to the solution of a general composite optimization problem of $\min_x f(x) + \kappa^*(x)$. Our findings provide valuable insights into the dynamics of Lion and pave the way for further improvements and extensions of Lion-related algorithms.

翻译：Lion（进化符号动量）是一种通过程序搜索发现的新型优化器，在训练大型人工智能模型方面展现出良好前景。其性能与 AdamW 相当或更优，同时具有更高的内存效率。正如随机搜索程序的结果所预期的那样，Lion 融合了多种现有算法的元素，包括符号动量、解耦权重衰减、Polak 动量和 Nesterov 动量，但无法归入任何现有理论优化器的范畴。因此，尽管 Lion 作为通用优化器在广泛任务中表现良好，其理论基础仍不明确。这种理论上的模糊性限制了对 Lion 效能进行进一步增强和扩展的机会。本研究旨在揭示 Lion 的理论本质。基于连续时间和离散时间分析，我们证明 Lion 是一种理论新颖且原理完备的方法，用于在施加边界约束 $\|x\|_\infty \leq 1/\lambda$ 的同时最小化一般损失函数 $f(x)$。Lion 通过引入解耦权重衰减实现这一目标，其中 $\lambda$ 表示权重衰减系数。我们的分析得益于为 Lion 更新设计的新型李雅普诺夫函数。该分析适用于更广泛的 Lion-$\kappa$ 算法族，其中 Lion 中的 $\text{sign}(\cdot)$ 算子被替换为凸函数 $\kappa$ 的次梯度，从而求解 $\min_x f(x) + \kappa^*(x)$ 的一般复合优化问题。我们的研究结果为理解 Lion 的动态特性提供了重要见解，并为 Lion 相关算法的进一步改进和扩展铺平了道路。