Mixture-of-Experts (MoE) models scale large language models efficiently by sparsely activating experts, but once an expert is selected, it is executed fully. Hence, the trade-off between accuracy and computation in an MoE model typically exhibits large discontinuities. We propose Mixture of Slimmable Experts (MoSE), an MoE architecture in which each expert has a nested, slimmable structure that can be executed at variable widths. This enables conditional computation not only over which experts are activated but also over how much of each expert is utilized. Consequently, a single pretrained MoSE model can support a more continuous spectrum of accuracy-compute trade-offs at inference time. We present a simple and stable training recipe for slimmable experts under sparse routing, combining multi-width training with standard MoE objectives. During inference, we explore strategies for runtime width determination, including a lightweight test-time training mechanism that learns how to map router confidence/probabilities to expert widths under a fixed budget. Experiments on GPT-style models, various routing regimes, zero-shot downstream reasoning benchmarks, and continual pre-training adaptation of DeepSeek model show that MoSE matches or improves standard MoE at full width and consistently shifts the compute-quality frontier toward lower inference FLOPs. The code can be found at: https://github.com/tnurbek/mose.

翻译：混合专家模型通过稀疏激活专家高效扩展大语言模型，但一旦专家被选中，其执行是完整的。因此，混合专家模型中准确率与计算量之间的权衡通常存在较大不连续性。我们提出混合可瘦身专家架构，其中每个专家具有嵌套的可瘦身结构，能够以可变宽度执行。这不仅实现了对激活哪些专家的条件计算，还能控制每个专家的利用程度。因此，单个预训练的MoSE模型在推理时可支持更连续的计算-准确率权衡谱系。我们提出一种简单且稳定的训练方案，在稀疏路由条件下结合多宽度训练与标准MoE目标函数来训练可瘦身专家。在推理阶段，我们探索运行时宽度确定策略，包括一种轻量级测试时训练机制，该机制学习如何在固定预算下将路由器置信度/概率映射到专家宽度。在GPT风格模型、多种路由策略、零样本下游推理基准以及DeepSeek模型的持续预训练适应实验表明，MoSE在满宽度下达到或超越标准MoE，并持续将计算质量前沿向更低推理FLOPs方向推进。代码可参见：https://github.com/tnurbek/mose。