Attention mechanisms are a central property of cognitive systems allowing them to selectively deploy cognitive resources in a flexible manner. Attention has been long studied in the neurosciences and there are numerous phenomenological models that try to capture its core properties. Recently attentional mechanisms have become a dominating architectural choice of machine learning and are the central innovation of Transformers. The dominant intuition and formalism underlying their development has drawn on ideas of keys and queries in database management systems. In this work, we propose an alternative Bayesian foundation for attentional mechanisms and show how this unifies different attentional architectures in machine learning. This formulation allows to to identify commonality across different attention ML architectures as well as suggest a bridge to those developed in neuroscience. We hope this work will guide more sophisticated intuitions into the key properties of attention architectures and suggest new ones.

翻译：注意机制是认知系统的一个核心特性，使其能够以灵活的方式选择性部署认知资源。神经科学领域对注意机制已有长期研究，并存在众多试图捕捉其核心特性的现象学模型。近期，注意机制已成为机器学习领域主导性的架构选择，并且是Transformer的核心创新。当前主导其发展的直觉与形式化框架源自数据库管理系统中的键值查询思想。在本工作中，我们提出了一种基于贝叶斯理论的替代性注意机制基础，并展示了其如何统一机器学习中的不同注意力架构。这一公式化方法使我们能够识别不同机器学习注意力架构的共性，并提出连接神经科学领域所发展模型的桥梁。我们希望本研究能引导对注意力架构关键特性更深入的直觉认知，并启发新的架构设计。