Large language models (LLMs) are versatile and can address many tasks, but for computational efficiency, it is often desirable to distill their capabilities into smaller student models. One way to do this for classification tasks is via dataset synthesis, which can be accomplished by generating examples of each label from the LLM. Prior approaches to synthesis use few-shot prompting, which relies on the LLM's parametric knowledge to generate usable examples. However, this leads to issues of repetition, bias towards popular entities, and stylistic differences from human text. In this work, we propose Synthesize by Retrieval and Refinement (SynthesizRR), which uses retrieval augmentation to introduce variety into the dataset synthesis process: as retrieved passages vary, the LLM is "seeded" with different content to generate its examples. We empirically study the synthesis of six datasets, covering topic classification, sentiment analysis, tone detection, and humor, requiring complex synthesis strategies. We find SynthesizRR greatly improves lexical and semantic diversity, similarity to human-written text, and distillation performance, when compared to standard 32-shot prompting and six baseline approaches.

翻译：大型语言模型（LLM）功能广泛，可处理多项任务，但出于计算效率考虑，通常需将其能力提炼至更小的学生模型中。对于分类任务，一种实现方式是通过数据集合成，即利用LLM生成每个标签的示例。以往的合成方法采用少样本提示，依赖LLM的参数化知识来生成可用示例。然而，这会导致重复性、对热门实体的偏见以及与人类文本的文体差异等问题。本文提出通过检索与精炼进行合成（SynthesizRR），该方法利用检索增强为数据集合成过程引入多样性：随着检索段落的变化，LLM被“播种”以不同内容来生成示例。我们实证研究了六个数据集的合成，涵盖主题分类、情感分析、语气检测和幽默，这些需要复杂的合成策略。结果发现，与标准的32次提示和六种基线方法相比，SynthesizRR显著提升了词汇和语义多样性、与人类书写文本的相似度以及蒸馏性能。