Understanding the locus of semantic representation in large language models (LLMs) is crucial for interpretability and architectural innovation. The dominant paradigm posits that trainable input embeddings serve as foundational "meaning vectors." This paper challenges that view. We construct Transformer models where the embedding layer is entirely frozen, with vectors derived not from data, but from the visual structure of Unicode glyphs. These non-semantic, precomputed visual embeddings are fixed throughout training. Our method is compatible with any tokenizer, including a novel Unicode-centric tokenizer we introduce to ensure universal text coverage. Despite the absence of trainable, semantically initialized embeddings, our models converge, generate coherent text, and, critically, outperform architecturally identical models with trainable embeddings on the MMLU reasoning benchmark. We attribute this to "representational interference" in conventional models, where the embedding layer is burdened with learning both structural and semantic features. Our results indicate that high-level semantics are not inherent to input embeddings but are an emergent property of the Transformer's compositional architecture and data scale. This reframes the role of embeddings from meaning containers to structural primitives. We release all code and models to foster further research.

翻译：理解大型语言模型（LLM）中语义表征的所在对于模型可解释性与架构创新至关重要。主流范式假定可训练的输入嵌入层是基础的“意义向量”。本文挑战了这一观点。我们构建了嵌入层完全冻结的Transformer模型，其向量并非源于数据，而是源自Unicode字符视觉结构。这些非语义的、预计算的视觉嵌入在训练过程中始终保持固定。我们的方法兼容任何分词器，包括我们提出的新型以Unicode为核心的分词器，以确保对任意文本的覆盖。尽管缺乏可训练的、具备语义初始化的嵌入层，我们的模型仍能收敛、生成连贯文本，并且关键的是，在MMLU推理基准测试中超越了架构相同但采用可训练嵌入层的模型。我们将此归因于传统模型中存在的“表征干扰”——嵌入层被迫同时学习结构特征与语义特征。我们的结果表明，高层语义并非输入嵌入的固有属性，而是Transformer组合架构与数据规模共同作用下的涌现特性。这重新定义了嵌入层的角色：从意义容器转变为结构基元。我们公开所有代码与模型以促进后续研究。