The in-context learning ability of large language models (LLMs) enables them to generalize to novel downstream tasks with relatively few labeled examples. However, they require enormous computational resources to be deployed. Alternatively, smaller models can solve specific tasks if fine-tuned with enough labeled examples. These examples, however, are expensive to obtain. In pursuit of the best of both worlds, we study synthetic data generation of fine-tuning training data via fine-tuned teacher LLMs to improve the downstream performance of much smaller models. In four text classification and two text generation tasks, we find that both data generation and annotation dramatically improve the respective downstream model's performance, occasionally necessitating only a minor fraction of the original training dataset.

翻译：大语言模型的上下文学习能力使其能够凭借相对较少的标注示例泛化到新下游任务。然而，其部署需要庞大的计算资源。相比之下，较小规模的模型若经充足标注样本微调，亦可解决特定任务，但获取这些样本的成本高昂。为兼顾两者优势，我们研究通过微调教师大语言模型生成合成训练数据的方法，以提升更小规模模型的下游性能。在四项文本分类任务与两项文本生成任务中，我们发现数据生成与标注均能显著提升对应下游模型的表现，有时仅需原始训练数据集的一小部分即可达成。