Recent studies have uncovered intriguing phenomena in deep learning, such as grokking, double descent, and emergent abilities in large language models, which challenge human intuition and are crucial for a deeper understanding of neural models. In this paper, we present a comprehensive framework that provides a unified view of these three phenomena, focusing on the competition between memorization and generalization circuits. This approach, initially employed to explain grokking, is extended in our work to encompass a wider range of model sizes and training data volumes. Our framework delineates four distinct training dynamics, each depending on varying combinations of model size and training data quantity. Utilizing this framework, we provide a detailed analysis of the double descent phenomenon and propose two verifiable predictions regarding its occurrence, both substantiated by our experimental results. Moreover, we expand our framework to the multi-task learning paradigm, demonstrating how algorithm tasks can be turned into emergent abilities. This offers a novel perspective to understand emergent abilities in Large Language Models.

翻译：近期研究揭示了深度学习中一些引人入胜的现象，如大语言模型中的Grokking、双下降和涌现能力，这些现象挑战了人类直觉，对于深入理解神经模型至关重要。本文提出了一个综合框架，以记忆电路与泛化电路之间的竞争为核心，为这三种现象提供了统一视角。该框架最初用于解释Grokking现象，后续在我们的工作中被扩展至更广泛的模型规模和训练数据量范围。我们的框架定义了四种不同的训练动力学模式，每种模式取决于模型规模与训练数据量的不同组合。基于此框架，我们详细分析了双下降现象，并提出了两项关于其发生条件的可验证预测，两项预测均得到实验结果支持。此外，我们将该框架扩展至多任务学习范式，展示了算法任务如何转化为涌现能力，从而为大语言模型中涌现能力的理解提供了全新视角。