We study dense and mixture-of-experts (MoE) transformers in a tiny-scale pretraining regime under a shared LLaMA-style decoder training recipe. The sparse model replaces dense feed-forward blocks with Mixtral-style routed experts. Dense baselines are modestly width-resized to tightly match either active or total parameter budgets, while tokenizer, data, optimizer, schedule, depth, context length, normalization style, and evaluation protocol are held fixed. Our best sparse recipe uses four experts, top-2 routing, Switch-style load balancing, and router z-loss. In a three-seed full-data comparison, the dense active-match model reaches 1.6545 +/- 0.0012 best validation loss, the MoE reaches 1.5788 +/- 0.0020, and the dense total-match model reaches 1.5608 +/- 0.0025. This yields a matched-active gap of 0.0758 +/- 0.0021 in the MoE's favor and a matched-total gap of 0.0180 +/- 0.0020 in the dense model's favor. Across training, the matched-active advantage grows while the matched-total dense advantage narrows sharply. In this sub-25M-parameter regime, MoE therefore improves validation loss under active-parameter matching but does not surpass dense training at equal total stored capacity.

翻译：我们研究在共享LLaMA风格解码器训练方案下，极小规模预训练场景中的密集和混合专家（MoE）变压器模型。稀疏网络采用Mixtral风格的专家路由替换密集前馈模块。密集基线模型通过适度调整宽度以紧密匹配活跃参数或总参数预算，而分词器、数据、优化器、学习率调度、深度、上下文长度、归一化风格及评估协议保持不变。我们最优的稀疏方案使用四位专家、top-2路由、Switch风格负载均衡及路由z-loss。在三种子全数据对比中，密集活跃匹配模型达到1.6545±0.0012的最佳验证损失，MoE达到1.5788±0.0020，密集总匹配模型达到1.5608±0.0025。这产生0.0758±0.0021的MoE优势（活跃参数匹配）和0.0180±0.0020的密集模型优势（总参数匹配）。训练过程中，活跃匹配优势持续增长，而总参数匹配下密集模型优势急剧收窄。在此低于2500万参数规模下，MoE在活跃参数匹配下能改善验证损失，但在总存储容量相等时未能超越密集训练。