Dot-product attention is a core module in the present generation of neural network models, particularly transformers, and is being leveraged across numerous areas such as natural language processing and computer vision. This attention module is comprised of three linear transformations, namely query, key, and value linear transformations, each of which has a bias term. In this work, we study the role of these bias terms, and mathematically show that the bias term of the key linear transformation is redundant and could be omitted without any impact on the attention module. Moreover, we argue that the bias term of the value linear transformation has a more prominent role than that of the bias term of the query linear transformation. We empirically verify these findings through multiple experiments on language modeling, natural language understanding, and natural language generation tasks.

翻译：点积注意力是当代神经网络模型（尤其是Transformer）的核心模块，已被广泛应用于自然语言处理和计算机视觉等多个领域。该注意力模块由三个线性变换组成，即查询、键和值线性变换，每个变换均包含偏置项。本研究探讨了这些偏置项的作用，并通过数学推导证明键线性变换中的偏置项是冗余的，移除该偏置项不会对注意力模块产生任何影响。此外，我们论证了值线性变换中的偏置项比查询线性变换中的偏置项具有更重要的作用。我们通过在语言建模、自然语言理解和自然语言生成任务上的多项实验，实证验证了上述发现。