We introduce GateSkip, a simple residual-stream gating mechanism that enables token-wise layer skipping in decoder-only LMs. Each Attention/MLP branch is equipped with a sigmoid-linear gate that condenses the branch's output before it re-enters the residual stream. During inference we rank tokens by the gate values and skip low-importance ones using a per-layer budget. While early-exit or router-based Mixture-of-Depths models are known to be unstable and need extensive retraining, our smooth, differentiable gates fine-tune stably on top of pretrained models. On long-form reasoning, we save up to 15% compute while retaining over 90% of baseline accuracy. For increasingly larger models, this tradeoff improves drastically. On instruction-tuned models we see accuracy gains at full compute and match baseline quality near 50% savings. The learned gates give insight into transformer information flow (e.g., BOS tokens act as anchors), and the method combines easily with quantization, pruning, and self-speculative decoding.

翻译：我们提出GateSkip，一种简单的残差流门控机制，可在仅解码器语言模型中实现基于词元的层跳过。每个注意力/多层感知机分支均配备一个sigmoid线性门，该门在分支输出重新进入残差流之前对其进行压缩。在推理过程中，我们根据门控值对词元进行排序，并依据每层计算预算跳过低重要性词元。虽然已知基于提前退出或路由器的混合深度模型存在不稳定性且需要大量重新训练，但我们提出的平滑可微分门控机制能够在预训练模型基础上稳定地进行微调。在长文本推理任务中，我们节省了高达15%的计算量，同时保持了超过90%的基线准确率。对于规模持续增大的模型，这种权衡效果显著提升。在指令微调模型中，我们在全计算量下观察到准确率提升，并在节省近50%计算量时仍能匹配基线质量。习得的门控机制为Transformer信息流提供了新的见解（例如BOS词元充当锚点），且该方法可轻松与量化、剪枝及自推测解码技术相结合。