The transformer neural network has significantly out-shined all other neural network architectures as the engine behind large language models. We provide a theoretical analysis of the expressivity of the transformer architecture through the lens of topos theory. From this viewpoint, we show that many common neural network architectures, such as the convolutional, recurrent and graph convolutional networks, can be embedded in a pretopos of piecewise-linear functions, but that the transformer necessarily lives in its topos completion. In particular, this suggests that the two network families instantiate different fragments of logic: the former are first order, whereas transformers are higher-order reasoners. Furthermore, we draw parallels with architecture search and gradient descent, integrating our analysis in the framework of cybernetic agents.

翻译：Transformer神经网络作为大型语言模型的核心引擎，已显著超越其他所有神经网络架构。我们通过拓扑斯理论的视角，对Transformer架构的表达能力进行了理论分析。从这一视角出发，我们表明：诸如卷积网络、循环网络和图卷积网络等常见神经网络架构，可嵌入分段线性函数的预拓扑斯中，而Transformer则必然存在于其拓扑斯完备化中。特别地，这表明两个网络家族实例化了不同的逻辑片段：前者为一阶逻辑，而Transformer为高阶推理器。此外，我们将其与架构搜索和梯度下降建立类比，并将分析纳入控制论智能体的框架。