Recent work claims that large language models display emergent abilities, abilities not present in smaller-scale models that are present in larger-scale models. What makes emergent abilities intriguing is two-fold: their sharpness, transitioning seemingly instantaneously from not present to present, and their unpredictability, appearing at seemingly unforeseeable model scales. Here, we present an alternative explanation for emergent abilities: that for a particular task and model family, when analyzing fixed model outputs, one can choose a metric which leads to the inference of an emergent ability or another metric which does not. Thus, our alternative suggests that existing claims of emergent abilities are creations of the researcher's analyses, not fundamental changes in model behavior on specific tasks with scale. We present our explanation in a simple mathematical model, then test it in three complementary ways: we (1) make, test and confirm three predictions on the effect of metric choice using the InstructGPT/GPT-3 family on tasks with claimed emergent abilities, (2) make, test and confirm two predictions about metric choices in a meta-analysis of emergent abilities on BIG-Bench; and (3) show how similar metric decisions suggest apparent emergent abilities on vision tasks in diverse deep network architectures (convolutional, autoencoder, transformers). In all three analyses, we find strong supporting evidence that emergent abilities may not be a fundamental property of scaling AI models.

翻译：近期研究声称大型语言模型展现出涌现能力，即那些在较小规模模型中不存在、而仅在更大规模模型中出现的特性。涌现能力的引人之处体现在两个方面：其一为突变性——似乎从不存在到存在瞬间转换；其二为不可预测性——在看似无法预见的模型规模下突然出现。本文对涌现能力提出另一种解释：针对特定任务与模型家族，当分析固定的模型输出时，研究者可选择能推导出涌现能力的评估指标，也可选择不产生这一结论的指标。因此，我们的替代性观点表明：现有关于涌现能力的论断是研究者分析方式的人为产物，而非模型在特定任务上随规模扩展而发生根本性行为变化。我们通过一个简洁的数学模型阐释该观点，并以三种互补方式进行验证：（1）针对标称具有涌现能力的任务，使用InstructGPT/GPT-3模型家族，对指标选择效应提出、检验并确认三项预测；（2）对BIG-Bench中涌现能力的元分析，提出、检验并确认两项关于指标选择的预测；（3）展示在视觉任务中，类似指标选择如何使不同深度网络架构（卷积网络、自编码器、Transformer）呈现出表观涌现能力。三项分析均获得有力证据支持：涌现能力可能并非扩展AI模型的基本属性。