In Transformer models, non-GEMM (non-General Matrix Multiplication) operations -- especially Softmax and Layer Normalization (LayerNorm) -- often dominate hardware cost due to their nonlinear nature. To address this, previous approximation studies mainly target rank-oriented tasks, which is acceptable for classification. However, edge Natural Language Processing (NLP) applications and edge generative AI are largely evaluated based on score-oriented tasks, so normalization-guaranteed non-GEMM operations are essential. We propose a hardware-efficient Softmax and LayerNorm with Guaranteed Normalization for Edge devices. Our design employs hardware-efficient approximation methods while preserving the normalization (Softmax: $\sum p = 1$, LayerNorm: $σ= 1$). Our architecture is described in Verilog HDL and synthesized using the Samsung 28nm CMOS process. In accuracy evaluation, we achieve high accuracy with minimal degradation: GLUE +0.07%, SQuAD -0.01%, perplexity -0.09%. Implementation results show that our architecture is small: $942\,μm^2$ for Softmax, $1199\,μm^2$ for LayerNorm. Compared to the state of the art, we achieve up to 11x and 14x reduction in area, respectively.

翻译：在Transformer模型中，非GEMM（非通用矩阵乘法）运算——尤其是Softmax和层归一化——因其非线性特性往往主导硬件成本。为应对此问题，先前的近似研究主要针对排序型任务，这在分类场景中可被接受。然而，边缘自然语言处理应用及边缘生成式人工智能主要基于评分型任务进行评估，因此具有归一化保证的非GEMM运算至关重要。我们提出一种面向边缘设备的硬件高效Softmax与层归一化方案，在采用硬件高效近似方法的同时保持归一化特性（Softmax：$\sum p = 1$，层归一化：$σ= 1$）。本架构采用Verilog HDL描述，并基于三星28nm CMOS工艺进行综合。精度评估表明，我们以最小退化实现高精度：GLUE指标提升+0.07%，SQuAD下降-0.01%，困惑度下降-0.09%。实现结果显示本架构面积紧凑：Softmax为$942\,μm^2$，层归一化为$1199\,μm^2$。与现有最优方案相比，本设计面积分别缩减高达11倍和14倍。