Pretrained large language models (LLMs) are currently state-of-the-art for solving the vast majority of natural language processing tasks. While many real-world applications still require fine-tuning to reach satisfactory levels of performance, many of them are in the low-data regime, making fine-tuning challenging. To address this, we propose LLM2LLM, a targeted and iterative data augmentation strategy that uses a teacher LLM to enhance a small seed dataset by augmenting additional data that can be used for fine-tuning on a specific task. LLM2LLM (1) fine-tunes a baseline student LLM on the initial seed data, (2) evaluates and extracts data points that the model gets wrong, and (3) uses a teacher LLM to generate synthetic data based on these incorrect data points, which are then added back into the training data. This approach amplifies the signal from incorrectly predicted data points by the LLM during training and reintegrates them into the dataset to focus on more challenging examples for the LLM. Our results show that LLM2LLM significantly enhances the performance of LLMs in the low-data regime, outperforming both traditional fine-tuning and other data augmentation baselines. LLM2LLM reduces the dependence on labor-intensive data curation and paves the way for more scalable and performant LLM solutions, allowing us to tackle data-constrained domains and tasks. We achieve improvements up to 24.2% on the GSM8K dataset, 32.6% on CaseHOLD, 32.0% on SNIPS, 52.6% on TREC and 39.8% on SST-2 over regular fine-tuning in the low-data regime using a LLaMA2-7B student model.

翻译：预训练大语言模型（LLMs）目前是解决绝大多数自然语言处理任务的最先进方法。然而，许多实际应用仍需微调才能达到满意的性能水平，且这些应用多处于低数据场景，使得微调面临挑战。为此，我们提出LLM2LLM——一种有针对性且迭代的数据增强策略，通过利用教师LLM增强小规模种子数据集，生成可用于特定任务微调的额外数据。LLM2LLM方法：（1）在初始种子数据上微调基线学生LLM；（2）评估并提取模型预测错误的数据点；（3）利用教师LLM基于这些错误数据点生成合成数据，并将其重新加入训练集。该方法能放大LLM训练过程中预测错误数据点的信号，通过将这些数据重新整合到数据集中，使模型更聚焦于学习困难样本。实验结果表明，LLM2LLM在低数据场景下显著提升了LLM的性能，优于传统微调方法及其他数据增强基线方法。该方法减少了对人工数据标注的依赖，为开发更具扩展性和高性能的LLM解决方案铺平了道路，使研究者能够应对数据受限的领域与任务。使用LLaMA2-7B学生模型时，在低数据场景下，我们的方法在GSM8K数据集上提升达24.2%，在CaseHOLD上提升32.6%，在SNIPS上提升32.0%，在TREC上提升52.6%，在SST-2上提升39.8%。