Despite recent advances in large language models, building dependable and deployable NLP models typically requires abundant, high-quality training data. However, task-specific data is not available for many use cases, and manually curating task-specific data is labor-intensive. Recent work has studied prompt-driven synthetic data generation using large language models, but these generated datasets tend to lack complexity and diversity. To address these limitations, we introduce a method, \textit{DataTune}, to make better use of existing, publicly available datasets to improve automatic dataset generation. DataTune performs dataset transformation, enabling the repurposing of publicly available datasets into a format that is directly aligned with the specific requirements of target tasks. On a diverse set of language-based tasks from the BIG-Bench benchmark, we find that finetuning language models via DataTune improves over a few-shot prompting baseline by 49\% and improves over existing methods that use synthetic or retrieved training data by 34\%. We find that dataset transformation significantly increases the diversity and difficulty of generated data on many tasks. We integrate DataTune into an open-source repository to make this method accessible to the community: https://github.com/neulab/prompt2model.

翻译：尽管大型语言模型近期取得了进展，构建可靠且可部署的NLP模型通常仍需要丰富且高质量的标注训练数据。然而，许多应用场景缺乏特定任务的数据，且人工标注任务数据需要大量劳动。近期研究探索了利用大型语言模型进行提示驱动的合成数据生成，但此类生成的数据集往往缺乏复杂性与多样性。为解决这些局限，我们提出一种名为\textit{DataTune}的方法，旨在更有效地利用现有公开数据集来改进自动化数据集生成。DataTune通过执行数据集转换，将公开数据集重塑为与目标任务需求直接对齐的格式。在BIG-Bench基准测试中涵盖的多种语言任务上，我们发现基于DataTune微调语言模型相比少样本提示基线提升49%，相比使用合成或检索训练数据的现有方法提升34%。研究表明，数据集转换显著提高了许多任务中生成数据的多样性与难度。我们将DataTune集成至开源仓库以便社区使用：https://github.com/neulab/prompt2model。