This paper presents a modular approach to accelerate inference in large language models (LLMs) by adding early exit heads at intermediate transformer layers. Each head is trained in a self-supervised manner to mimic the main model's predictions, allowing computation to stop early when a calibrated confidence threshold is reached. We evaluate several confidence metrics and show that entropy provides the most reliable separation between correct and incorrect predictions. Experiments on the Pythia model suite (70M to 2.8B parameters) demonstrate that our method significantly reduces inference cost while maintaining accuracy across multiple benchmarks. We further adapt this approach to speculative decoding, introducing Dynamic Self-Speculative Decoding (DSSD), which achieves 1.66x higher token acceptance than manually-tuned LayerSkip baselines with minimal hyperparameter tuning.

翻译：本文提出一种模块化方法，通过在Transformer中间层添加早期退出头来加速大型语言模型推理。每个退出头以自监督方式训练，以模拟主模型的预测结果，当达到校准后的置信度阈值时可提前终止计算。我们评估了多种置信度度量指标，证明熵能为正确与错误预测提供最可靠的区分度。在Pythia模型套件上的实验表明，该方法在保持多项基准测试精度的同时，显著降低了推理成本。我们进一步将该方法适配至推测解码场景，提出动态自推测解码方法，在最小化超参数调整的前提下，其词元接受率比人工调优的LayerSkip基线提高了1.66倍。