Decoding by contrasting layers (DoLa), is designed to improve the generation quality of large language models (LLMs) by contrasting the prediction probabilities between an early exit output (amateur logits) and the final output (expert logits). However, we find that this approach does not work well on non-English tasks. Inspired by previous interpretability work on language transition during the model's forward pass, we discover that this issue arises from a language mismatch between early exit output and final output. In this work, we propose an improved contrastive decoding algorithm that is effective for diverse languages beyond English. To obtain more helpful amateur logits, we devise two strategies to skip a set of bottom, language-agnostic layers based on our preliminary analysis. Experimental results on multilingual reasoning benchmarks demonstrate that our proposed method outperforms previous contrastive decoding baselines and substantially improves LLM's chain-of-thought reasoning accuracy across 11 languages. The project will be available at: https://github.com/NJUNLP/SkipLayerCD.

翻译：层间对比解码（DoLa）旨在通过对比早期退出输出（业余对数概率）与最终输出（专家对数概率）之间的预测概率，来提升大语言模型（LLM）的生成质量。然而，我们发现该方法在非英语任务上效果不佳。受先前关于模型前向传播过程中语言转换可解释性研究的启发，我们发现该问题源于早期退出输出与最终输出之间的语言不匹配。在本工作中，我们提出了一种改进的对比解码算法，该算法对英语之外的多种语言均有效。为了获得更有帮助的业余对数概率，基于初步分析，我们设计了两种策略来跳过一组底层的语言无关层。在多语言推理基准测试上的实验结果表明，我们提出的方法优于先前的对比解码基线，并显著提升了LLM在11种语言上的思维链推理准确率。项目地址：https://github.com/NJUNLP/SkipLayerCD。