Mixture-of-experts (MoE) layers enable the scaling of transformer models while keeping the inference compute fixed. While task-expert specialization has been observed in empirical studies of frontier MoE transformer models, existing theoretical work analyzes this using continuous mixture models that cannot be used to model natural language effectively. An important open question is to \textit{theoretically explain task-expert specialization in transformer MoE models using discrete models of language}. To address this, we represent structured knowledge via syntactic templates and finite key-value dictionaries, and prove formally that a single-layer MoE transformer can encode knowledge by using experts that specialize in the corresponding tasks. Our construction shows how queries are routed to unique, task-specific experts whose size depends solely on the intrinsic complexity of the given task (i.e. the combined size of its syntactic templates and factual dictionary). Our construction provides a theoretical support for empirical results on localized knowledge circuits in MoE models. We support our theoretical findings with experiments evaluating model performance under varying MoE loss functions.

翻译：混合专家层在保持推理计算量不变的情况下实现了Transformer模型的扩展。虽然前沿混合专家Transformer模型的实证研究已观察到任务-专家特化现象，但现有理论工作使用连续混合模型进行分析，无法有效建模自然语言。一个重要悬而未决的问题是：**利用离散语言模型从理论上解释Transformer混合专家模型中的任务-专家特化**。为此，我们通过句法模板和有限键值字典来表示结构化知识，并严格证明单层混合专家Transformer可通过使用专精于对应任务的专家来编码知识。我们的构造展示了查询如何被路由至唯一的任务特异性专家，其规模完全取决于给定任务的内在复杂度（即其句法模板与事实字典的联合规模）。该构造为混合专家模型中局部化知识电路的实证结果提供了理论支撑。我们通过实验评估不同混合专家损失函数下模型性能，进一步支持了理论发现。