Progressive Learning (PL) reduces pre-training computational overhead by gradually increasing model scale. While prior work has extensively explored depth expansion, width expansion remains significantly understudied, with the few existing methods limited to the early stages of training. However, expanding width during the mid-stage is essential for maximizing computational savings, yet it remains a formidable challenge due to severe training instabilities. Empirically, we show that naive initialization at this stage disrupts activation statistics, triggering loss spikes, while copy-based initialization introduces gradient symmetry that hinders feature diversity. To address these issues, we propose SPARKLING (balancing {S}ignal {P}reservation {A}nd symmet{R}y brea{K}ing for width-progressive {L}earn{ING}), a novel framework for mid-stage width expansion. Our method achieves signal preservation via RMS-scale consistency, stabilizing activation statistics during expansion. Symmetry breaking is ensured through asymmetric optimizer state resetting and learning rate re-warmup. Extensive experiments on Mixture-of-Experts (MoE) models demonstrate that, across multiple width axes and optimizer families, SPARKLING consistently outperforms training from scratch and reduces training cost by up to 35% under $2\times$ width expansion.

翻译：渐进式学习（Progressive Learning, PL）通过逐步增加模型规模来减少预训练的计算开销。虽然先前的工作已广泛探索了深度扩展，但宽度扩展的研究仍显著不足，现有的少数方法也仅限于训练的早期阶段。然而，在训练中期进行宽度扩展对于最大化计算节省至关重要，但由于严重的训练不稳定性，这仍然是一个巨大的挑战。我们通过实证表明，在此阶段进行简单初始化会破坏激活统计量，引发损失尖峰，而基于复制的初始化则会引入梯度对称性，从而阻碍特征多样性。为解决这些问题，我们提出了SPARKLING（平衡宽度渐进式学习中的信号保持与对称性破缺），这是一个用于训练中期宽度扩展的新颖框架。我们的方法通过RMS尺度一致性实现信号保持，从而在扩展过程中稳定激活统计量。对称性破缺则通过非对称优化器状态重置和学习率重新预热来确保。在混合专家（Mixture-of-Experts, MoE）模型上进行的大量实验表明，在多个宽度维度和优化器族中，SPARKLING始终优于从头开始的训练，并在$2\times$宽度扩展下将训练成本降低高达35%。