In this paper, we introduce Parallel Attention and Feed-Forward Net Design (PAF) for transformer models. Transformer models are indisputably the backbone of all Natural Language Processing applications. Therefore, any efforts aimed at improving their efficiency are guaranteed to have an enormous impact. Transformer models consist of many layers and each layer has an attention block followed by a feed-forward network (FFN) that processes the input based on the attention block's output. We refer to this standard design as Series Attention and Feed-Forward Net Design (SAF). For each layer in our proposed PAF design for transformer models, we make FFN block's computations independent of the output of the attention block. This decoupling allows FFN block of each layer to run in parallel to the attention block of that layer. We evaluate PAF design by training two large language models (RoBERTa-large and bert-large-uncased) and comparing them to their SAF counterparts on six tasks of the General Language Understanding (GLUE) benchmark which test a multitude of semantic attributes. PAF models achieves nearly identical performance as their SAF counterparts on all the six tasks. We also compare time complexities of attention blocks with FFN blocks and find that running both blocks in parallel can theoretically and in practice achieve upto 1.5x to 2x gains in speed. We leave the development of fast and efficient libraries for implementation of PAF design for future work.

翻译：本文提出面向Transformer模型的并行注意力与前馈网络设计（PAF）。Transformer模型无疑是所有自然语言处理应用的基石，因此任何旨在提升其效率的工作都必将产生巨大影响。Transformer模型由多层构成，每层包含一个注意力模块和一个前馈网络（FFN），FFN基于注意力模块的输出处理输入。我们将这种标准设计称为串行注意力与前馈网络设计（SAF）。在我们提出的PAF设计中，每层的FFN模块计算独立于该层注意力模块的输出。这种解耦使得每层的FFN模块可以与注意力模块并行运行。我们通过训练两个大语言模型（RoBERTa-large和bert-large-uncased）来评估PAF设计，并在通用语言理解（GLUE）基准测试中测试多种语义属性的六个任务上将其与对应的SAF模型进行比较。PAF模型在所有六个任务上均实现了与SAF模型几乎一致的性能。我们还比较了注意力模块与FFN模块的时间复杂度，发现并行运行两个模块在理论和实践中均可实现高达1.5倍至2倍的速度提升。我们将开发用于实现PAF设计的快速高效库的工作留待未来研究。