Several previous works concluded that the largest part of generation capabilities of large language models (LLM) are learned (early) during pre-training. However, LLMs still require further alignment to adhere to downstream task requirements and stylistic preferences, among other desired properties. As LLMs continue to scale in terms of size, the computational cost of alignment procedures increase prohibitively. In this work, we propose a novel approach to circumvent these costs via proxy-based test-time alignment, i.e. using guidance from a small aligned model. Our approach can be described as a token-specific cascading method, where the token-specific deferral rule is reduced to 0-1 knapsack problem. In this setting, we derive primal and dual approximations of the optimal deferral decision. We experimentally show the benefits of our method both in task performance and speculative decoding speed.

翻译：多项先前研究指出，大型语言模型（LLM）的大部分生成能力是在预训练阶段（早期）习得的。然而，LLM仍需进一步对齐以适应下游任务需求、风格偏好及其他期望特性。随着LLM规模持续扩大，对齐过程的计算成本呈指数级增长。本研究提出一种创新方法，通过基于代理的测试时对齐（即利用小型对齐模型的引导）规避这些成本。该方法可描述为一种面向特定令牌的级联方法，其中令牌级延迟决策被简化为0-1背包问题。在此框架下，我们推导出最优延迟决策的原始近似解与对偶近似解。实验结果表明，该方法在任务性能和推测解码速度方面均展现出显著优势。