Mixture-of-Experts (MoE) architectures efficiently scale Large Language Models (LLMs) by activating only a small fraction of their experts per token, yet the full parameter count - dominated by the expert parameters - must be held in training and inference memory. To address this, we introduce Expert Tying, an architectural modification that shares expert parameters across consecutive transformer layers while preserving independent, layer-wise routing and attention. We evaluate this approach across common, state-of-the-art architectures, including OLMoE, Qwen3, and DeepSeek-style MoEs. Our pretraining experiments demonstrate that tying experts can reduce memory footprint by almost 2x at virtually no degradation in perplexity or downstream quality. By exploiting the parameter redundancy inherent in MoE pathways, our method provides a highly favorable compute-to-memory trade-off, advancing efficient training and scaling of next-generation LLMs.

翻译：混合专家（MoE）架构通过每个令牌仅激活少量专家，实现了大规模语言模型（LLM）的高效扩展，然而，训练和推理内存中必须保存全部参数（其中专家参数占据主导）。为解决此问题，我们提出专家共享技术，这是一种架构改进方法，在保持独立逐层路由与注意力机制的前提下，跨连续Transformer层共享专家参数。我们在包括OLMoE、Qwen3和DeepSeek风格MoE在内的通用前沿架构上评估了该方法。预训练实验表明，专家共享能在几乎不降低困惑度或下游任务质量的情况下，将内存占用减少近2倍。通过利用MoE路径固有的参数冗余，我们的方法提供了极为有利的计算-内存权衡，推动下一代LLM的高效训练与扩展。