Aligning large language models (LLMs) with human preferences is essential for their applications. Recently, decoding-time alignment has emerged as an effective plug-and-play technique that avoids fine-tuning model parameters. This approach retains the general utility of pretrained LLMs but often suffers from significant inefficiencies during decoding, primarily due to wasted token generation and excessive reward evaluations. To address these challenges, we introduce Cascade Reward Sampling (CARDS) to resolve both efficiency bottlenecks in decoding-time alignment. Specifically, we develop a segment-level rejection sampling algorithm that minimizes redundant computations of both LLMs and reward models (RMs). Central to CARDS is an uncertainty-based segmentation mechanism, which ensures the accuracy of RMs evaluations on incomplete segments. Furthermore, we provide a detailed analysis of reward scores on segments to elucidate the improved alignment performance. Experimental results demonstrate that CARDS significantly improves decoding efficiency, alignment quality, and general utility compared to existing decoding-time alignment methods, achieving approximately a 70% reduction in decoding time and over 90% win-ties in utility and safety benchmarks.

翻译：使大语言模型（LLM）与人类偏好对齐对其应用至关重要。近年来，解码时对齐已成为一种有效的即插即用技术，无需微调模型参数。该方法保留了预训练大语言模型的通用能力，但在解码过程中常因无效的令牌生成和过度的奖励评估而导致显著效率低下。为应对这些挑战，我们提出了级联奖励采样（CARDS），以解决解码时对齐中的两个效率瓶颈。具体而言，我们开发了一种分段级拒绝采样算法，最大限度地减少了大语言模型和奖励模型（RM）的冗余计算。CARDS的核心是一种基于不确定性的分段机制，该机制确保了奖励模型对不完整片段评估的准确性。此外，我们对片段上的奖励分数进行了详细分析，以阐明其改进的对齐性能。实验结果表明，与现有的解码时对齐方法相比，CARDS显著提升了解码效率、对齐质量和通用能力，实现了约70%的解码时间缩减，并在效用与安全性基准测试中取得了超过90%的胜平率。