Large language models (LLMs) with instruction fine-tuning demonstrate superior generative capabilities. However, these models are resource-intensive. To alleviate this issue, we explore distilling knowledge from instruction-tuned LLMs into much smaller ones. To this end, we carefully develop a large set of 2.58M instructions based on both existing and newly-generated instructions. In addition to being sizable, we design our instructions to cover a broad set of topics to ensure diversity. Extensive analysis of our instruction dataset confirms its diversity, and we generate responses for these instructions using gpt-3.5-turbo. Leveraging these instructions, we fine-tune a diverse herd of models, collectively referred to as LaMini-LM, which includes models from both the encoder-decoder and decoder-only families, with varying sizes. We evaluate the performance of our models using automatic metrics on 15 different natural language processing (NLP) benchmarks, as well as through human assessment. The results demonstrate that our proposed LaMini-LM models are comparable to competitive baselines, while being much smaller in size.

翻译：经过指令微调的大语言模型展现出卓越的生成能力。然而，这些模型对计算资源需求巨大。为解决这一问题，我们探索将指令微调后的大语言模型的知识蒸馏到更小的模型中。为此，我们基于现有指令和新生成的指令精心构建了一个包含258万条指令的大规模数据集。除了规模庞大外，我们设计的指令覆盖了广泛的主题以确保多样性。对该指令数据集的深入分析证实了其多样性，我们使用gpt-3.5-turbo为这些指令生成响应。利用这些指令，我们对涵盖编码器-解码器和仅解码器两大架构家族、不同规模的多样化模型群进行微调，统称为LaMini-LM。我们通过15个自然语言处理基准任务的自动指标评估以及人工评估来验证模型性能。结果表明，提出的LaMini-LM模型虽体积更小，但性能与竞争性基准模型不相上下。