We consider the problem of eliciting compositional generalization capabilities in large language models (LLMs) with a novel type of prompting strategy. Compositional generalization empowers the LLMs to solve problems that are harder than the ones they have seen (i.e., easy-to-hard generalization), which is a critical reasoning capability of human-like intelligence. However, even the current state-of-the-art LLMs still struggle with this form of reasoning. To bridge this gap, we propose skills-in-context (SKiC) prompting, which instructs LLMs how to compose basic skills to resolve more complex problems. We find that it is crucial to demonstrate both the skills and the compositional examples within the same prompting context. With as few as two examplars, our SKiC prompting initiates strong synergies between skills and their composition capabilities. Notably, it empowers LLMs to solve unseen problems that require innovative skill compositions, achieving near-perfect generalization on a broad range of challenging compositionality tasks. Intriguingly, SKiC prompting unlocks the latent potential of LLMs, enabling them to leverage pre-existing internal skills acquired during earlier pretraining and alignment stages, even when these skills are not explicitly presented in the prompting context. This results in the capability of LLMs to solve unseen complex problems by activating and composing these internal competencies.

翻译：我们考虑通过一种新型提示策略来激发大语言模型（LLMs）的组合泛化能力。组合泛化使LLMs能够解决比其已见问题更难的问题（即从易到难的泛化），这是类人智能的关键推理能力。然而，即使是当前最先进的LLMs仍难以掌握这种推理形式。为填补这一空白，我们提出技能上下文（SKiC）提示方法，该方法指导LLMs如何将基础技能组合以解决更复杂的问题。研究发现，在同一提示上下文中同时展示技能和组合示例至关重要。仅需两个示例，我们的SKiC提示就能在技能与组合能力之间建立强协同效应。值得注意的是，它使LLMs能够解决需要创新技能组合的未见问题，在广泛且富有挑战性的组合任务上实现了近乎完美的泛化。耐人寻味的是，SKiC提示解锁了LLMs的潜在能力，使其能够利用早期预训练和对齐阶段获得的内在既有技能——即便这些技能未被显式呈现于提示上下文中。这赋予了LLMs通过激活并组合这些内部能力来解决未见复杂问题的能力。