While Mixture-of-Experts (MoE) scales model capacity without proportionally increasing computation, its massive total parameter footprint creates significant storage and memory-access bottlenecks, which hinder efficient end-side deployment that simultaneously requires high performance, low computational cost, and small storage overhead. To achieve these properties, we present DECO, a sparse MoE architecture designed to match the performance of dense Transformers under identical total parameter budgets and training tokens. DECO utilizes the differentiable and flexible ReLU-based routing enhanced by learnable expert-wise scaling, which adaptively balances the contributions of routed and shared experts. Furthermore, we introduce NormSiLU, an activation function that normalizes inputs prior to SiLU operators, producing a more stable trend of routed-expert activation ratio and a higher intrinsic sparsity level. We also identify an empirical advantage in using non-gated MLP experts with ReLU-based routing, indicating the possibility of MoE architecture simplification. Experiments demonstrate that DECO, activating only 20% of routed experts, matches dense performance and outperforms established MoE baselines. Our specialized acceleration kernel delivers a 2.93$\times$ speedup on Jetson AGX Orin compared with dense inference. Code and checkpoints are available at https://github.com/thunlp/DECO.

翻译：混合专家（MoE）模型虽能在不按比例增加计算量的前提下扩展模型容量，但其庞大的总参数量造成了显著的存储和内存访问瓶颈，阻碍了同时要求高性能、低计算开销和低存储负担的端侧高效部署。为实现这些特性，我们提出DECO——一种稀疏MoE架构，旨在相同总参数量和训练数据量下达到与稠密Transformer相当的性能。DECO采用基于可微柔性ReLU的路由机制，并通过可学习的专家级缩放进行增强，从而自适应地平衡路由专家与共享专家的贡献。此外，我们引入NormSiLU激活函数，该函数在SiLU算子之前对输入进行归一化处理，使路由专家的激活比例呈现更稳定的趋势，并实现更高的固有稀疏度。我们还发现，在基于ReLU的路由中使用非门控MLP专家具有实证优势，这暗示了MoE架构简化的可能性。实验表明，DECO仅激活20%的路由专家即可达到与稠密模型相当的性能，且优于现有MoE基线模型。我们的专用加速核在Jetson AGX Orin上相较于稠密推理实现了2.93倍加速。代码与检查点已开源至https://github.com/thunlp/DECO。