Mixture-of-Experts (MoE) architectures scale large language models (LLMs) by activating only a subset of experts per token, but the standard TopK routing assigns the same fixed number of experts to all tokens, ignoring their varying complexity. Prior adaptive routing methods introduce additional modules and hyperparameters, often requiring costly retraining from scratch. We propose Sequence-level TopK (SeqTopK), a minimal modification that shifts the expert budget from the token level to the sequence level. By selecting the top $T \cdot K$ experts across all $T$ tokens, SeqTopK enables end-to-end learned dynamic allocation -- assigning more experts to difficult tokens and fewer to easy ones -- while preserving the same overall budget. SeqTopK requires only a few lines of code, adds less than 1% overhead, and remains fully compatible with pretrained MoE models. Experiments across math, coding, law, and writing show consistent improvements over TopK and prior parameter-free adaptive methods, with gains that become substantially larger under higher sparsity (up to 16.9%). These results highlight SeqTopK as a simple, efficient, and scalable routing strategy, particularly well-suited for the extreme sparsity regimes of next-generation LLMs. Code is available at https://github.com/Y-Research-SBU/SeqTopK.

翻译：混合专家（MoE）架构通过仅激活每个词元的部分专家来扩展大型语言模型（LLM），但标准TopK路由为所有词元分配固定的相同专家数量，忽略了其复杂度差异。既有自适应路由方法引入了额外模块和超参数，通常需要从头开始进行高昂成本的重训练。我们提出序列级TopK（SeqTopK），这是一种最小化修改，将专家预算从词元级转移至序列级。通过在所有$T$个词元中选择排名前$T \cdot K$的专家，SeqTopK实现了端到端学习的动态分配——为困难词元分配更多专家，为简单词元分配更少专家——同时保持整体预算不变。SeqTopK仅需几行代码，额外开销低于1%，且与预训练MoE模型完全兼容。在数学、编程、法律和写作领域的实验表明，SeqTopK相较于TopK及先前无参数自适应方法持续取得改进，且在更高稀疏度下增益显著增大（最高达16.9%）。这些结果凸显SeqTopK作为一种简单、高效且可扩展的路由策略，尤其适用于下一代LLM的极端稀疏场景。代码已开源至https://github.com/Y-Research-SBU/SeqTopK。