While Mixture-of-Experts (MoE) architectures have become the standard for sparsity scaling in large language models, they increasingly face diminishing returns and system-level bottlenecks. In this work, we explore embedding scaling as a potent, orthogonal dimension for scaling sparsity. Through a comprehensive analysis and experiments, we identify specific regimes where embedding scaling achieves a superior Pareto frontier compared to expert scaling. We systematically characterize the critical architectural factors governing this efficacy -- ranging from parameter budgeting to the interplay with model width and depth. Moreover, by integrating tailored system optimizations and speculative decoding, we effectively convert this sparsity into tangible inference speedups. Guided by these insights, we introduce LongCat-Flash-Lite, a 68.5B parameter model with ~3B activated trained from scratch. Despite allocating over 30B parameters to embeddings, LongCat-Flash-Lite not only surpasses parameter-equivalent MoE baselines but also exhibits exceptional competitiveness against existing models of comparable scale, particularly in agentic and coding domains.

翻译：尽管专家混合架构已成为大型语言模型中稀疏性扩展的标准方法，但其正面临收益递减和系统级瓶颈日益加剧的问题。在本研究中，我们探索将嵌入层缩放作为一种强大且正交的稀疏性扩展维度。通过全面分析与实验，我们识别出嵌入层缩放相较于专家模块缩放能够实现更优帕累托前沿的具体机制。我们系统性地揭示了决定该效能的关键架构因素——涵盖参数预算分配、与模型宽度及深度的交互作用等方面。此外，通过整合定制化系统优化与推测解码技术，我们成功将这种稀疏性转化为可观的推理加速效果。基于这些发现，我们提出了从头训练的LongCat-Flash-Lite模型，该模型具有685亿参数且每次激活约30亿参数。尽管为嵌入层分配了超过300亿参数，LongCat-Flash-Lite不仅在同等参数量下超越了专家混合基线模型，更在智能体与代码生成领域展现出相较于同规模现有模型的卓越竞争力。