Symbol grounding (Harnad, 1990) describes how symbols such as words acquire their meanings by connecting to real-world sensorimotor experiences. Recent work has shown preliminary evidence that grounding may emerge in (vision-)language models trained at scale without using explicit grounding objectives. Yet, the specific loci of this emergence and the mechanisms that drive it remain largely unexplored. To address this problem, we introduce a controlled evaluation framework that systematically traces how symbol grounding arises within the internal computations through mechanistic and causal analysis. Our findings show that grounding concentrates in middle-layer computations and is implemented through the aggregate mechanism, where attention heads aggregate the environmental ground to support the prediction of linguistic forms. This phenomenon replicates in multimodal dialogue and across architectures (Transformers and state-space models), but not in unidirectional LSTMs. Our results provide behavioral and mechanistic evidence that symbol grounding can emerge in language models, with practical implications for predicting and potentially controlling the reliability of generation.

翻译：符号奠基（Harnad, 1990）描述了词语等符号如何通过与真实世界的感觉运动经验相连接来获得其意义。近期研究初步表明，在未使用显式奠基目标的情况下，大规模训练的（视觉-）语言模型中可能涌现出符号奠基现象。然而，这种涌现的具体位点及其驱动机制在很大程度上仍未得到探索。为解决该问题，我们引入了一个受控评估框架，通过机制性分析与因果分析系统性地追踪符号奠基如何在内部计算过程中产生。研究结果表明，符号奠基集中于中间层计算，并通过聚合机制实现——其中注意力头聚合环境依据以支持语言形式的预测。该现象在多模态对话场景及不同架构（Transformer与状态空间模型）中均可复现，但单向LSTM中未观察到。我们的结果提供了行为学与机制性证据，证明语言模型中能够涌现符号奠基，这对预测乃至控制生成结果的可靠性具有实际意义。