We study dense-to-sparse continual training as a way to construct channel-sparse large language models from dense checkpoints. Starting from a Qwen2.5-8B dense backbone, we continue training at 32K context and introduce a predictor-gated sparse SwiGLU FFN in the 32K stage. For each token and layer, we use a low-rank predictor to produce FFN-channel routing logits. We then apply a bank-wise top-k rule to retain 16 channels in every 64-channel bank, yielding 4x sparsity in the FFN intermediate activation. Unlike post-hoc sparse inference methods, the routing module is placed on the main language modeling path and optimized during continual training, enabling the dense model to be upcycled into a hardware-oriented sparse model. We report the architecture, training recipe, benchmark performance, and training lessons. We also identify a layer-local long-context failure mode on RULER-CWE and propose a single-layer repair algorithm that substantially improves the affected length range.

翻译：我们研究将稠密到稀疏的持续训练作为从稠密检查点构建通道稀疏大语言模型的方法。从Qwen2.5-8B稠密主干网络出发，我们在32K上下文下持续训练，并在32K阶段引入预测门控稀疏SwiGLU前馈网络（FFN）。对于每个词元（token）和层，我们使用低秩预测器生成FFN通道路由logits，随后应用基于分库（bank-wise）的top-k规则，在每64个通道的库中保留16个通道，从而在FFN中间激活中实现4倍稀疏性。与事后稀疏推理方法不同，路由模块被置于主语言建模路径上，并在持续训练过程中进行优化，使得稠密模型能够升级为面向硬件的稀疏模型。我们报告了模型架构、训练方案、基准性能及训练经验。同时，我们在RULER-CWE任务上发现了一种逐层长上下文故障模式，并提出了一种单层修复算法，显著改善了受影响的长度范围。