Large Language Models (LLMs) demonstrate remarkable performance on a variety of Natural Language Understanding (NLU) tasks, primarily due to their in-context learning ability. This ability is utilized in our proposed "CoThought" pipeline, which efficiently trains smaller "baby" language models (BabyLMs) by leveraging the Chain of Thought (CoT) prompting of LLMs. Our pipeline restructures a dataset of less than 100M in size using GPT-3.5-turbo, transforming it into task-oriented, human-readable texts that are comparable to the school texts for language learners. The BabyLM is then pretrained on this restructured dataset in a RoBERTa (Liu et al., 2019) fashion. In evaluations across 4 benchmarks, our BabyLM outperforms the RoBERTa-base in 10 linguistic, NLU, and question answering tasks by more than 3 points, showing superior ability to extract contextual information. These results suggest that compact LMs pretrained on small, LLM-restructured data can better understand tasks and achieve improved performance. The code for data processing and model training is available at: https://github.com/oooranz/Baby-CoThought.

翻译：大型语言模型（LLMs）在各种自然语言理解任务中展现出卓越性能，这主要归功于它们的上下文学习能力。我们提出的"CoThought"流水线充分利用了这一能力，通过LLMs的思维链提示高效训练更小的"婴儿"语言模型。该流水线使用GPT-3.5-turbo重构一个规模不足1亿条的数据集，将其转化为面向任务的、人类可读的文本，这类文本堪比语言学习者的教材文本。随后，采用RoBERTa（Liu等，2019）的方式，在此重构数据集上对婴儿语言模型进行预训练。在4个基准测试的评估中，我们的婴儿语言模型在10项语言学、自然语言理解和问答任务上，均以超过3个百分点的优势超越RoBERTa-base模型，展现出更强的上下文信息提取能力。这些结果表明，在基于LLM重构的小规模数据上预训练的紧凑型语言模型，能够更好地理解任务并实现性能提升。数据处理和模型训练的代码可在以下地址获取：https://github.com/oooranz/Baby-CoThought。