We consider the problem of eliciting compositional generalization capabilities in large language models (LLMs) with a novel type of prompting strategy. Compositional generalization empowers the LLMs to solve problems that are harder than the ones they have seen (i.e., easy-to-hard generalization), which is a critical reasoning capability of human-like intelligence. However, even the current state-of-the-art LLMs still struggle with this form of reasoning. To bridge this gap, we propose skills-in-context (SKiC) prompting, which instructs LLMs how to compose basic skills to resolve more complex problems. We find that it is crucial to demonstrate both the skills and the compositional examples within the same prompting context. With as few as two examplars, our SKiC prompting initiates strong synergies between skills and their composition capabilities. Notably, it empowers LLMs to solve unseen problems that require innovative skill compositions, achieving near-perfect generalization on a broad range of challenging compositionality tasks. Intriguingly, SKiC prompting unlocks the latent potential of LLMs, enabling them to leverage pre-existing internal skills acquired during earlier pre-training stages, even when these skills are not explicitly presented in the prompting context. This results in the capability of LLMs to solve unseen complex problems by activating and composing internal competencies. With such prominent features, SKiC prompting is able to achieve state-of-the-art performance on challenging mathematical reasoning benchmarks (e.g., MATH).

翻译：我们研究了一种新型提示策略，旨在激发大型语言模型（LLMs）的组合泛化能力。组合泛化使LLMs能够解决比已见过问题更复杂的难题（即从易到难的泛化），这是类人智能的关键推理能力。然而，当前最先进的LLMs仍难以掌握这种推理形式。为弥补这一差距，我们提出了技能情境（SKiC）提示方法，该方法引导LLMs如何组合基本技能以解决更复杂的问题。我们发现，在同一提示情境中同时展示技能及组合示例至关重要。仅需两个样例，SKiC提示就能在技能与其组合能力之间建立强大的协同效应。值得注意的是，它使LLMs能够解决需要创新技能组合的未见问题，在一系列具有挑战性的组合任务上实现近乎完美的泛化。有趣的是，SKiC提示解锁了LLMs的潜在能力，使其能够利用早期预训练阶段获取的既有内部技能——即使这些技能未在提示情境中明确呈现。这使LLMs能够通过激活并组合内部能力来解决复杂的未见问题。凭借这些突出特性，SKiC提示在具有挑战性的数学推理基准测试（如MATH）上实现了最先进的性能。