The transformer neural network architecture uses a form of attention in which the dot product of query and key is divided by the square root of the key dimension before applying softmax. This scaling of the dot product is designed to avoid the absolute value of the dot products becoming so large that applying softmax leads to vanishing gradients. In this paper, we propose some alternative scalings, including dividing the dot product instead by the sum of the key lengths before applying softmax. We use simulated keys and queries to show that in many situations this appears to be more effective at avoiding regions where applying softmax leads to vanishing gradients.

翻译：Transformer 神经网络架构采用一种注意力机制，其在应用 softmax 之前，将查询与键的点积除以键维度的平方根。这种点积缩放旨在避免点积绝对值过大而导致 softmax 产生梯度消失。本文提出了一些替代缩放方案，包括在应用 softmax 前将点积除以键长度之和。我们通过模拟键与查询证明，在许多情况下，该方法在避免 softmax 导致梯度消失的区域方面更为有效。