The attention mechanism is a pivotal element within the Transformer architecture, making a substantial contribution to its exceptional performance. Within this attention mechanism, Softmax is an imperative component that enables the model to assess the degree of correlation between various segments of the input. Yet, prior research has shown that Softmax operations can significantly increase processing latency and energy consumption in the Transformer network due to their internal nonlinear operations and data dependencies. In this work, we proposed~\textit{Hyft}, a hardware efficient floating point Softmax accelerator for both training and inference. Hyft aims to reduce the implementation cost of different nonlinear arithmetic operations by adaptively converting intermediate results into the most suitable numeric format for each specific operation, leading to reconfigurable accelerator with hybrid numeric format. The evaluation results highlight that Hyft achieves a remarkable $15\times$ reduction in hardware resource utilization and a $20 \times$ reduction in processing latency, all while maintaining a negligible impact on Transformer accuracy.

翻译：注意力机制是Transformer架构中的关键要素，为其卓越性能做出了重要贡献。在该注意力机制中，Softmax作为不可或缺的组件，使模型能够评估输入各部分之间的关联程度。然而，已有研究表明，由于Softmax运算包含内部非线性操作和数据依赖，其会显著增加Transformer网络的处理延迟和能耗。本文提出了\textit{Hyft}——一种面向训练和推理的高效硬件浮点Softmax加速器。Hyft通过将中间结果自适应地转换为每种特定运算最合适的数值格式，降低不同非线性算术运算的实现成本，从而构建具有混合数值格式的可重构加速器。评估结果表明，Hyft在硬件资源利用率上实现了显著的$15\times$降低，处理延迟减少$20 \times$，同时对Transformer精度的影响可忽略不计。