Neuromorphic or neurally-inspired optimizers rely on local but parallel parameter updates to solve problems that range from quadratic programming to Ising machines. An ideal realization of such an optimizer not only uses a compute-in-memory (CIM) paradigm to address the so-called memory-wall (i.e. energy dissipated due to repeated memory read access), but also uses a learning-in-memory (LIM) paradigm to address the energy bottlenecks due to repeated memory writes at the precision required for optimization (the update-wall), and to address the energy bottleneck due to the repeated transfer of information between short-term and long-term memories (the consolidation-wall). In this paper, we derive theoretical estimates for the energy-to-solution metric that can be achieved by this ideal neuromorphic optimizer which is realized by modulating the energy-barrier of the physical memories such that the dynamics of memory updates and memory consolidation matches the optimization or the annealing dynamics. The analysis presented in this paper captures the out-of-equilibrium thermodynamics of learning and the resulting energy-efficiency estimates are model-agnostic which only depend on the number of model-update operations (OPS), the model-size in terms of number of parameters, the speed of convergence, and the precision of the solution. To show the practical applicability of our results, we apply our analysis for estimating the lower-bound on the energy-to-solution metrics for large-scale AI workloads.

翻译：神经形态或神经启发的优化器依赖局部但并行的参数更新来解决从二次规划到伊辛机的一系列问题。此类优化器的理想实现不仅采用内存计算范式以解决所谓的"内存墙"问题（即因重复内存读取访问导致的能量耗散），还采用学习内存范式以解决优化所需精度下重复内存写入导致的能量瓶颈（更新墙），以及短期记忆与长期记忆间信息重复传输导致的能量瓶颈（巩固墙）。本文推导了这种理想神经形态优化器所能达到的解能效指标的理论估计，该优化器通过调节物理存储器的能垒实现，使得存储器更新与巩固的动态过程匹配优化或退火动态。本文分析捕捉了学习的非平衡态热力学特性，所得能效估计具有模型无关性，仅取决于模型更新操作次数、参数数量表示的模型规模、收敛速度及求解精度。为展示研究结果的实际适用性，我们将分析应用于大规模人工智能工作负载的解能效指标下界估计。