Transformer models gain popularity because of their superior inference accuracy and inference throughput. However, the transformer is computation-intensive, causing a long inference time. The existing works on transformer inference acceleration have limitations caused by either the modification of transformer architectures or the need of specialized hardware. In this paper, we identify the opportunities of using memoization to accelerate the self-attention mechanism in transformers without the above limitations. Built upon a unique observation that there is rich similarity in attention computation across inference sequences, we build a memoization database that leverages the emerging big memory system. We introduce a novel embedding technique to find semantically similar inputs to identify computation similarity. We also introduce a series of techniques such as memory mapping and selective memoization to avoid memory copy and unnecessary overhead. We enable 22% inference-latency reduction on average (up to 68%) with negligible loss in inference accuracy.

翻译：Transformer模型因其优越的推理精度和推理吞吐量而日益流行。然而，Transformer属于计算密集型模型，导致推理时间较长。现有的Transformer推理加速方法存在局限性，其原因是要么需要修改Transformer架构，要么需要专用硬件。本文中，我们发现了利用记忆化加速Transformer中自注意力机制的机会，从而避免了上述局限性。基于对推理序列间注意力计算存在丰富相似性的独特观察，我们构建了一个利用新兴大内存系统的记忆化数据库。我们引入了一种新颖的嵌入技术来寻找语义相似的输入，以识别计算相似性。我们还引入了一系列技术，如内存映射和选择性记忆化，以避免内存复制和不必要的开销。我们实现了平均22%（最高68%）的推理延迟降低，且推理精度损失可忽略不计。