Recent work targeting large language models (LLMs) for code generation demonstrated that increasing the amount of training data through synthetic code generation often leads to exceptional performance. In this paper we explore data pruning methods aimed at enhancing the efficiency of model training specifically for code LLMs. We present techniques that integrate various clustering and pruning metrics to selectively reduce training data without compromising the accuracy and functionality of the generated code. We observe significant redundancies in synthetic training data generation, where our experiments demonstrate that benchmark performance can be largely preserved by training on only 10% of the data. Moreover, we observe consistent improvements in benchmark results through moderate pruning of the training data. Our experiments show that these pruning strategies not only reduce the computational resources needed but also enhance the overall quality code generation.

翻译：近期针对代码生成的大语言模型（LLM）研究表明，通过合成代码生成增加训练数据量通常能带来卓越的性能。本文探索旨在提升代码LLM训练效率的数据剪枝方法。我们提出了融合多种聚类与剪枝指标的技术，以选择性减少训练数据，同时不损害生成代码的准确性与功能性。我们观察到合成训练数据生成中存在显著冗余，实验表明仅使用10%的数据进行训练即可基本保持基准性能。此外，通过对训练数据进行适度剪枝，我们在基准测试结果中观察到持续改进。实验证明，这些剪枝策略不仅能降低所需计算资源，还能提升代码生成的整体质量。