Knowledge editing (KE) enables precise modifications to factual content in large language models (LLMs). Existing KE methods are largely designed for dense architectures, limiting their applicability to the increasingly prevalent sparse Mixture-of-Experts (MoE) models that underpin modern scalable LLMs. Although MoEs offer strong efficiency and capacity scaling, naively adapting dense-model editors is both computationally costly and prone to routing distribution shifts that undermine stability and consistency. To address these challenges, we introduce MoEEdit, the first routing-stable framework for parameter-modifying knowledge editing in MoE LLMs. Our method reparameterizes expert updates via per-expert null-space projections that keep router inputs invariant and thereby suppress routing shifts. The resulting block-structured optimization is solved efficiently with a block coordinate descent (BCD) solver. Experiments show that MoEEdit attains state-of-the-art efficacy and generalization while preserving high specificity and routing stability, with superior compute and memory efficiency. These results establish a robust foundation for scalable, precise knowledge editing in sparse LLMs and underscore the importance of routing-stable interventions.
翻译:知识编辑能够对大语言模型中的事实内容进行精确修改。现有知识编辑方法主要针对稠密架构设计,限制了其在日益流行的稀疏专家混合模型中的应用,而后者正是现代可扩展大语言模型的基础。尽管MoE模型具备优异的效率与容量扩展能力,但直接适配稠密模型编辑器不仅计算成本高昂,且易引发路由分布偏移,从而破坏编辑的稳定性与一致性。为应对这些挑战,我们提出了MoEEdit——首个面向MoE大语言模型参数修改型知识编辑的路由稳定框架。该方法通过专家级零空间投影对专家更新进行重参数化,保持路由器输入不变从而抑制路由偏移。由此产生的块结构优化问题可通过块坐标下降求解器高效求解。实验表明,MoEEdit在保持高特异性与路由稳定性的同时,实现了最优的编辑效能与泛化能力,并具备卓越的计算与内存效率。这些结果为稀疏大语言模型的可扩展精确知识编辑奠定了坚实基础,并凸显了路由稳定干预机制的重要性。