Learning-in-memory (LIM) is a recently proposed paradigm to overcome fundamental memory bottlenecks in training machine learning systems. While compute-in-memory (CIM) approaches can address the so-called memory-wall (i.e. energy dissipated due to repeated memory read access) they are agnostic to the energy dissipated due to repeated memory writes at the precision required for training (the update-wall), and they don't account for the energy dissipated when transferring information between short-term and long-term memories (the consolidation-wall). The LIM paradigm proposes that these bottlenecks, too, can be overcome if the energy barrier of physical memories is adaptively modulated such that the dynamics of memory updates and consolidation match the Lyapunov dynamics of gradient-descent training of an AI model. In this paper, we derive new theoretical lower bounds on energy dissipation when training AI systems using different LIM approaches. The analysis presented here is model-agnostic and highlights the trade-off between energy efficiency and the speed of training. The resulting non-equilibrium energy-efficiency bounds have a similar flavor as that of Landauer's energy-dissipation bounds. We also extend these limits by taking into account the number of floating-point operations (FLOPs) used for training, the size of the AI model, and the precision of the training parameters. Our projections suggest that the energy-dissipation lower-bound to train a brain scale AI system (comprising of $10^{15}$ parameters) using LIM is $10^8 \sim 10^9$ Joules, which is on the same magnitude the Landauer's adiabatic lower-bound and $6$ to $7$ orders of magnitude lower than the projections obtained using state-of-the-art AI accelerator hardware lower-bounds.

翻译：内存学习(LIM)是一种新近提出的范式，旨在克服机器学习系统训练中的根本性内存瓶颈。尽管存内计算(CIM)方法能够解决所谓的内存墙问题（即因重复内存读取所耗散的能耗），但它们无法顾及训练所需精度下因重复内存写入所耗散的能耗（更新墙问题），也未考虑短时记忆与长时记忆之间信息传输所耗散的能耗（固结墙问题）。LIM范式提出，若能自适应地调节物理内存的能量势垒，使内存更新与固结的动态过程匹配AI模型梯度下降训练的Lyapunov动力学特征，这些瓶颈同样可以克服。本文推导了采用不同LIM方法训练AI系统时能耗的理论新下界。本文提出的分析是模型无关的，并揭示了能效与训练速度之间的权衡关系。所得的非平衡能效界限与兰道尔能耗界限具有相似性质。我们还通过考虑训练所使用的浮点运算次数(FLOPs)、AI模型规模以及训练参数精度，对这些极限进行了拓展。我们的预测表明，使用LIM训练一个包含$10^{15}$个参数的类脑规模AI系统的能耗下界为$10^8 \sim 10^9$焦耳，这与兰道尔绝热下界处于同一数量级，且比基于最先进AI加速器硬件下界得到的预测值低6至7个数量级。