Large Language Models have become the de facto approach to sequence-to-sequence text generation tasks, but for specialized tasks/domains, a pretrained LLM lacks specific capabilities to produce accurate or well-formatted responses. Supervised fine-tuning specializes a LLM by training it on dataset of example prompts with target responses, but real-world data tends to be noisy. While many fine-tuning algorithms exist, here we consider a \emph{data-centric AI} perspective on LLM fine-tuning, studying how to \emph{systematically} curate the training dataset to improve the LLM produced via \emph{any} fine-tuning algorithm. We introduce an automated data curation pipeline CLEAR (Confidence-based LLM Evaluation And Rectification) for instruction tuning datasets, that can be used with any LLM and fine-tuning procedure. CLEAR estimates which training data is low-quality and either filters or corrects it. Automatically identifying which data to filter or correct is done via LLM-derived confidence estimates, to ensure only confident modifications to the dataset. Unlike existing data curation techniques, CLEAR is a comprehensive framework that can improve a dataset (and trained model outputs) without additional fine-tuning computations. We don't assume access to a stronger LLM than the model being fine-tuned (e.g.\ relying on GPT-4 when fine-tuning GPT-3.5), to see whether CLEAR can meaningfully improve the capabilities of any LLM. Experiments reveal that CLEAR consistently improves the performance of fine-tuned models across many datasets and models (like GPT-3.5 and Llama2).

翻译：大型语言模型已成为序列到序列文本生成任务的事实标准方法，但在特定任务或领域上，预训练的语言模型缺乏产生准确或格式良好的响应所需的具体能力。监督微调通过在包含示例提示和目标响应的数据集上训练来专门化语言模型，但真实世界的数据往往存在噪声。尽管存在多种微调算法，本文从数据中心的视角研究语言模型的微调，探讨如何系统地策展训练数据集以改进通过任何微调算法得到的语言模型。我们提出了一种名为CLEAR（基于置信度的语言模型评估与修正）的自动化数据策展流水线，适用于指令微调数据集，可与任何语言模型及微调过程配合使用。CLEAR评估哪些训练数据质量较低，并对其进行过滤或修正。自动识别需要过滤或修正的数据通过语言模型导出的置信度估计实现，以确保仅对数据集进行可信的修改。与现有数据策展技术不同，CLEAR是一个无需额外微调计算即可改进数据集（及训练后模型输出）的综合框架。我们假设在测试CLEAR能否有效提升任意语言模型能力时，无法访问强于被微调模型的语言模型（例如微调GPT-3.5时依赖GPT-4）。实验表明，CLEAR在多个数据集和模型（如GPT-3.5和Llama2）上持续提升了微调模型的性能。