The self-attention mechanism sets transformer-based large language model (LLM) apart from the convolutional and recurrent neural networks. Despite the performance improvement, achieving real-time LLM inference on silicon is challenging due to the extensively used Softmax in self-attention. Apart from the non-linearity, the low arithmetic intensity greatly reduces the processing parallelism, which becomes the bottleneck especially when dealing with a longer context. To address this challenge, we propose Constant Softmax (ConSmax), a software-hardware co-design as an efficient Softmax alternative. ConSmax employs differentiable normalization parameters to remove the maximum searching and denominator summation in Softmax. It allows for massive parallelization while performing the critical tasks of Softmax. In addition, a scalable ConSmax hardware utilizing a bitwidth-split look-up table (LUT) can produce lossless non-linear operation and support mix-precision computing. It further facilitates efficient LLM inference. Experimental results show that ConSmax achieves a minuscule power consumption of 0.43 mW and area of 0.001 mm2 at 1-GHz working frequency and 22-nm CMOS technology. Compared to state-of-the-art Softmax hardware, ConSmax results in 14.5x energy and 14.0x area savings with a comparable accuracy on a GPT-2 model and the WikiText103 dataset.

翻译：自注意力机制使基于Transformer的大语言模型（LLM）区别于卷积神经网络和递归神经网络。尽管性能有所提升，但由于自注意力机制中广泛使用Softmax，在硅基芯片上实现LLM实时推理仍具有挑战性。除非线性特性外，低算术强度大幅降低了处理并行性，这尤其在处理长上下文时成为瓶颈。为解决这一挑战，我们提出常数Softmax（ConSmax），一种作为高效Softmax替代方案的软硬件协同设计方法。ConSmax采用可微归一化参数，无需执行Softmax中的最大值搜索和分母求和，在完成Softmax关键任务的同时支持大规模并行化。此外，利用位宽分离查找表（LUT）的可扩展ConSmax硬件，可实现无损非线性运算并支持混合精度计算，进一步促进高效LLM推理。实验结果表明，在1GHz工作频率和22nm CMOS工艺下，ConSmax功耗仅为0.43mW，面积为0.001mm²。与最先进的Softmax硬件相比，在GPT-2模型和WikiText103数据集上保持相当精度的前提下，ConSmax实现了14.5倍的能耗节约和14.0倍的面积节约。