Analogy is a central faculty of human intelligence, enabling abstract patterns discovered in one domain to be applied to another. Despite its central role in cognition, the mechanisms by which Transformers acquire and implement analogical reasoning remain poorly understood. In this work, inspired by the notion of functors in category theory, we formalize analogical reasoning as the inference of correspondences between entities across categories. Based on this formulation, we introduce synthetic tasks that evaluate the emergence of analogical reasoning under controlled settings. We find that the emergence of analogical reasoning is highly sensitive to data characteristics, optimization choices, and model scale. Through mechanistic analysis, we show that analogical reasoning in Transformers decomposes into two key components: (1) geometric alignment of relational structure in the embedding space, and (2) the application of a functor within the Transformer. These mechanisms enable models to transfer relational structure from one category to another, realizing analogy. Finally, we quantify these effects and find that the same trends are observed in pretrained LLMs. In doing so, we move analogy from an abstract cognitive notion to a concrete, mechanistically grounded phenomenon in modern neural networks.

翻译：类比是人类智能的核心能力，它使得在一个领域发现的抽象模式能够应用于另一个领域。尽管类比在认知中扮演着核心角色，但Transformer模型如何习得并实现类比推理的机制仍不甚明晰。受范畴论中函子概念的启发，本研究将类比推理形式化为跨范畴实体间对应关系的推断。基于此形式化框架，我们设计了一系列合成任务，用于评估受控环境下类比推理能力的涌现现象。研究发现，类比推理的涌现对数据特征、优化策略及模型规模高度敏感。通过机制分析，我们揭示了Transformer中的类比推理可分解为两个关键组成部分：(1) 嵌入空间中关系结构的几何对齐，以及(2) Transformer内部函子的应用。这些机制使得模型能够将关系结构从一个范畴迁移到另一个范畴，从而实现类比。最后，我们量化了这些效应，并发现相同的趋势在预训练大语言模型中同样存在。通过这项工作，我们将类比从抽象的认知概念转化为现代神经网络中具体且具有机制基础的现象。