Although large language models (LLMs) have advanced the state-of-the-art in NLP significantly, deploying them for downstream applications is still challenging due to cost, responsiveness, control, or concerns around privacy and security. As such, trainable models are still the preferred option in some cases. However, these models still require human-labeled data for optimal performance, which is expensive and time-consuming to obtain. In order to address this issue, several techniques to reduce human effort involve labeling or generating data using LLMs. Although these methods are effective for certain applications, in practice they encounter difficulties in real-world scenarios. Labeling data requires careful data selection, while generating data necessitates task-specific prompt engineering. In this paper, we propose a unified data creation pipeline that requires only a single formatting example, and which is applicable to a broad range of tasks, including traditionally problematic ones with semantically devoid label spaces. In our experiments we demonstrate that instruction-following LLMs are highly cost-effective data creators, and that models trained with these data exhibit performance better than those trained with human-labeled data (by up to 17.5%) on out-of-distribution evaluation, while maintaining comparable performance on in-distribution tasks. These results have important implications for the robustness of NLP systems deployed in the real-world.

翻译：尽管大型语言模型（LLM）显著提升了自然语言处理（NLP）的最新技术水平，但由于成本、响应速度、可控性以及隐私和安全方面的考虑，将其部署到下游应用仍然面临挑战。因此，在某些情况下，可训练模型仍然是首选方案。然而，这些模型仍需依赖人工标注数据才能实现最优性能，而获取此类数据既昂贵又耗时。为解决这一问题，一些减少人工参与的技术采用了利用LLM进行数据标注或生成的方法。尽管这些方法对特定应用有效，但在实际场景中仍会遇到困难：数据标注需要精心选择数据，而数据生成则需要针对具体任务设计提示。本文提出了一种统一的数据创建流程，仅需单个格式化示例，即可适用于广泛任务，包括传统上因标签空间缺乏语义区分而难以处理的任务。我们的实验表明，遵循指令的LLM是极具成本效益的数据创造者，使用这些数据训练的模型在分布外评估中性能优于使用人工标注数据训练的模型（最高提升17.5%），同时在分布内任务中保持相当的性能。这些结果对部署于现实世界的NLP系统鲁棒性具有重要启示意义。