Modern large language model (LLM) alignment techniques rely on human feedback, but it is unclear whether these techniques fundamentally limit the capabilities of aligned LLMs. In particular, it is unknown if it is possible to align (stronger) LLMs with superhuman capabilities with (weaker) human feedback without degrading their capabilities. This is an instance of the weak-to-strong generalization problem: using feedback from a weaker (less capable) model to train a stronger (more capable) model. We prove that weak-to-strong generalization is possible by eliciting latent knowledge from pre-trained LLMs. In particular, we cast the weak-to-strong generalization problem as a transfer learning problem in which we wish to transfer a latent concept prior from a weak model to a strong pre-trained model. We prove that a naive fine-tuning approach suffers from fundamental limitations, but an alternative refinement-based approach suggested by the problem structure provably overcomes the limitations of fine-tuning. Finally, we demonstrate the practical applicability of the refinement approach in multiple LLM alignment tasks.

翻译：现代大型语言模型（LLM）对齐技术依赖于人类反馈，但尚不清楚这些技术是否从根本上限制了已对齐LLM的能力。特别是，我们不知道是否可能在不降低其能力的情况下，利用（较弱的）人类反馈来对齐具有超人类能力的（更强的）LLM。这是弱到强泛化问题的一个实例：利用来自较弱（能力较低）模型的反馈来训练较强（能力较高）的模型。我们证明了通过从预训练的LLM中提取潜在知识，弱到强泛化是可能的。具体而言，我们将弱到强泛化问题表述为一个迁移学习问题，其中我们希望将潜在概念先验从弱模型迁移到强预训练模型。我们证明了朴素的微调方法存在根本性局限，但由问题结构启发的另一种基于精炼的方法在理论上能够克服微调的局限性。最后，我们在多个LLM对齐任务中展示了精炼方法的实际适用性。