In-context learning, i.e., learning from context examples, is an impressive ability of Transformer. Training Transformers to possess this in-context learning skill is computationally intensive due to the occurrence of learning plateaus, which are periods within the training process where there is minimal or no enhancement in the model's in-context learning capability. To study the mechanism behind the learning plateaus, we conceptually seperate a component within the model's internal representation that is exclusively affected by the model's weights. We call this the "weights component", and the remainder is identified as the "context component". By conducting meticulous and controlled experiments on synthetic tasks, we note that the persistence of learning plateaus correlates with compromised functionality of the weights component. Recognizing the impaired performance of the weights component as a fundamental behavior drives learning plateaus, we have developed three strategies to expedite the learning of Transformers. The effectiveness of these strategies is further confirmed in natural language processing tasks. In conclusion, our research demonstrates the feasibility of cultivating a powerful in-context learning ability within AI systems in an eco-friendly manner.

翻译：上下文学习，即从上下文示例中学习，是Transformer的一项令人印象深刻的能力。训练Transformer获得这种上下文学习技能由于学习平台期的出现而计算密集，这些平台期是训练过程中模型上下文学习能力几乎没有或完全没有提升的阶段。为研究学习平台期背后的机制，我们在概念上将模型内部表示中仅受模型权重影响的部分分离出来，称之为"权重分量"，其余部分则定义为"上下文分量"。通过在合成任务上进行细致且受控的实验，我们注意到学习平台期的持续存在与权重分量功能受损相关。认识到权重分量性能受损是驱动学习平台期的基本行为，我们开发了三种加速Transformer学习的策略。这些策略的有效性在自然语言处理任务中得到了进一步证实。总之，我们的研究证明了以生态友好方式在人工智能系统中培养强大上下文学习能力的可行性。