Large Language Models (LLMs) demonstrate remarkable performance on a variety of natural language understanding (NLU) tasks, primarily due to their in-context learning ability. This ability could be applied to building babylike models, i.e. models at small scales, improving training efficiency. In this paper, we propose a "CoThought" pipeline, which efficiently trains smaller "baby" language models (BabyLMs) by leveraging the Chain of Thought prompting of LLMs. Our pipeline restructures a dataset of less than 100M in size using GPT-3.5-turbo, transforming it into task-oriented, human-readable texts that are comparable to the school texts for language learners. The BabyLM is then pretrained on this restructured dataset in a RoBERTa fashion. In evaluations across 4 benchmarks, our BabyLM outperforms the vanilla RoBERTa in 10 linguistic, NLU, and question-answering tasks by more than 3 points, showing a superior ability to extract contextual information. These results suggest that compact LMs pretrained on small, LLM-restructured data can better understand tasks and achieve improved performance.

翻译：大型语言模型（LLMs）在多种自然语言理解（NLU）任务中展现出卓越性能，这主要归功于它们的上下文学习能力。该能力可应用于构建婴儿级模型（即小规模模型），从而提升训练效率。本文提出一种"协思"（CoThought）流水线，通过利用LLMs的思维链提示，高效训练较小的"婴儿"语言模型（BabyLMs）。我们的流水线使用GPT-3.5-turbo重构了一个规模不足1亿的数据集，将其转化为面向任务、可读性强的文本，相当于语言学习者使用的教材文本。随后采用RoBERTa方式对该重构数据集进行BabyLM的预训练。在4项基准测试中，我们的BabyLM在10项语言学、NLU和问答任务上均优于标准RoBERTa模型超过3个百分点，展现出更强的上下文信息提取能力。这些结果表明，基于LLM重构的小规模数据预训练的紧凑语言模型，能够更好地理解任务并实现性能提升。