The self-attention mechanism sets transformer-based large language model (LLM) apart from the convolutional and recurrent neural networks. Despite the performance improvement, achieving real-time LLM inference on silicon is challenging due to the extensively used Softmax in self-attention. Apart from the non-linearity, the low arithmetic intensity greatly reduces the processing parallelism, which becomes the bottleneck especially when dealing with a longer context. To address this challenge, we propose Constant Softmax (ConSmax), a software-hardware co-design as an efficient Softmax alternative. ConSmax employs differentiable normalization parameters to remove the maximum searching and denominator summation in Softmax. It allows for massive parallelization while performing the critical tasks of Softmax. In addition, a scalable ConSmax hardware utilizing a bitwidth-split look-up table (LUT) can produce lossless non-linear operation and support mix-precision computing. It further facilitates efficient LLM inference. Experimental results show that ConSmax achieves a minuscule power consumption of 0.43 mW and area of 0.001 mm2 at 1-GHz working frequency and 22-nm CMOS technology. Compared to state-of-the-art Softmax hardware, ConSmax results in 14.5x energy and 14.0x area savings with a comparable accuracy on a GPT-2 model and the WikiText103 dataset.

翻译：自注意力机制使得基于Transformer的大语言模型（LLM）区别于卷积神经网络和循环神经网络。尽管性能有所提升，但由于自注意力机制中广泛使用的Softmax，在硅基芯片上实现实时LLM推理仍面临挑战。除了非线性特性外，低算术强度大幅降低了处理并行性，这尤其在处理长上下文时成为瓶颈。针对这一挑战，我们提出常数Softmax（ConSmax），一种软件-硬件协同设计的、高效的Softmax替代方案。ConSmax采用可微归一化参数，省去Softmax中的最大值搜索和分母求和步骤，在完成Softmax关键任务的同时实现大规模并行化。此外，利用位宽分割查找表（LUT）的可扩展ConSmax硬件可实现无损非线性运算并支持混合精度计算，进一步促进高效的LLM推理。实验结果表明，在1 GHz工作频率和22纳米CMOS工艺下，ConSmax的功耗仅为0.43 mW，面积为0.001 mm²。与最先进的Softmax硬件相比，ConSmax在GPT-2模型与WikiText103数据集上保持相近精度的前提下，实现了14.5倍的能耗节省和14.0倍的面积节省。