Natural Language Processing (NLP) models optimized for predictive performance often make high confidence errors and suffer from vulnerability to adversarial and out-of-distribution data. Existing work has mainly focused on mitigation of such errors using either humans or an automated approach. In this study, we explore the usage of large language models (LLMs) for data augmentation as a potential solution to the issue of NLP models making wrong predictions with high confidence during classification tasks. We compare the effectiveness of synthetic data generated by LLMs with that of human data obtained via the same procedure. For mitigation, humans or LLMs provide natural language characterizations of high confidence misclassifications to generate synthetic data, which are then used to extend the training set. We conduct an extensive evaluation of our approach on three classification tasks and demonstrate its effectiveness in reducing the number of high confidence misclassifications present in the model, all while maintaining the same level of accuracy. Moreover, we find that the cost gap between humans and LLMs surpasses an order of magnitude, as LLMs attain human-like performance while being more scalable.

翻译：自然语言处理（NLP）模型在优化预测性能时，常对高置信度错误预测表现出脆弱性，且易受对抗性和分布外数据的影响。现有研究主要依赖人工或自动化方法缓解此类错误。本研究探索使用大语言模型（LLM）进行数据增强，作为解决NLP模型在分类任务中高置信度错误预测问题的潜在方案。我们比较了LLM生成的合成数据与通过相同流程获取的人工数据的有效性。在缓解策略中，人类或LLM通过提供高置信度错误分类的自然语言描述生成合成数据，并将其用于扩展训练集。我们在三个分类任务上对该方法进行了广泛评估，结果表明，该方法能在保持相同准确率水平的同时，有效减少模型中的高置信度错误分类数量。此外，我们发现人类与LLM之间的成本差距超过一个数量级，因为LLM在实现类似人类性能的同时更具可扩展性。