Modern language models (LMs) can learn to perform new tasks in different ways: in instruction following, the target task is described explicitly in natural language; in few-shot prompting, the task is specified implicitly with a small number of examples; in instruction inference, LMs are presented with in-context examples and are then prompted to generate a natural language task description before making predictions. Each of these procedures may be thought of as invoking a different form of reasoning: instruction following involves deductive reasoning, few-shot prompting involves inductive reasoning, and instruction inference involves abductive reasoning. How do these different capabilities relate? Across four LMs (from the gpt and llama families) and two learning problems (involving arithmetic functions and machine translation) we find a strong dissociation between the different types of reasoning: LMs can sometimes learn effectively from few-shot prompts even when they are unable to explain their own prediction rules; conversely, they sometimes infer useful task descriptions while completely failing to learn from human-generated descriptions of the same task. Our results highlight the non-systematic nature of reasoning even in some of today's largest LMs, and underscore the fact that very different learning mechanisms may be invoked by seemingly similar prompting procedures.

翻译：现代语言模型能够以不同方式习得执行新任务：在指令遵循中，目标任务通过自然语言显式描述；在少样本提示中，任务通过少量示例隐式指定；在指令推理中，模型先获得上下文示例，随后被要求生成自然语言任务描述再进行预测。这些过程可分别对应不同推理形式：指令遵循涉及演绎推理，少样本提示涉及归纳推理，指令推理涉及溯因推理。这些不同能力之间存在怎样的关联？通过对四种语言模型（来自GPT与LLaMA系列）及两类学习问题（涉及算术函数与机器翻译）的研究，我们发现不同推理类型之间存在显著分离：语言模型有时能从少样本提示中有效学习，却无法解释自身的预测规则；反之，它们有时能推断出有用的任务描述，却完全无法从人类生成的任务描述中学习。我们的结果揭示了即使在当前某些最大规模的语言模型中，推理仍存在非系统性特征，并强调了看似相似的提示过程可能调用截然不同的学习机制这一事实。