Recent work demonstrates that, after being fine-tuned on a high-quality instruction dataset, the resulting model can obtain impressive capabilities to address a wide range of tasks. However, existing methods for instruction data generation often produce duplicate data and are not controllable enough on data quality. In this paper, we extend the generalization of instruction tuning by classifying the instruction data to 4 code-related tasks and propose a LLM-based Generator-Discriminator data process framework to generate diverse, high-quality instruction data from open source code. Hence, we introduce CodeOcean, a dataset comprising 20,000 instruction instances across 4 universal code-related tasks,which is aimed at augmenting the effectiveness of instruction tuning and improving the generalization ability of fine-tuned model. Subsequently, we present WaveCoder, a fine-tuned Code LLM with Widespread And Versatile Enhanced instruction tuning. This model is specifically designed for enhancing instruction tuning of Code Language Models (LLMs). Our experiments demonstrate that Wavecoder models outperform other open-source models in terms of generalization ability across different code-related tasks at the same level of fine-tuning scale. Moreover, Wavecoder exhibits high efficiency in previous code generation tasks. This paper thus offers a significant contribution to the field of instruction data generation and fine-tuning models, providing new insights and tools for enhancing performance in code-related tasks.

翻译：近期研究表明，经过高质量指令数据集微调后，模型能够展现出处理广泛任务的出色能力。然而，现有指令数据生成方法常产生重复数据，且对数据质量的可控性不足。本文通过将指令数据划分为四类代码相关任务，拓展了指令微调的泛化能力，并提出一种基于大语言模型的生成器-判别器数据处理框架，旨在从开源代码中生成多样化、高质量的指令数据。为此，我们构建了CodeOcean数据集，包含20，000个覆盖四类通用代码相关任务的指令实例，旨在提升指令微调的有效性并增强微调模型的泛化能力。进而，我们提出WaveCoder——一种通过广泛而全面的增强指令微调获得的代码大语言模型。该模型专为增强代码语言模型的指令微调而设计。实验表明，在相同微调规模下，WaveCoder模型在不同代码相关任务的泛化能力上优于其他开源模型。此外，WaveCoder在既有代码生成任务中展现出高效性。本文为指令数据生成与模型微调领域做出了重要贡献，为提升代码相关任务性能提供了新思路与实用工具。