Writing commit messages is a tedious daily task for many software developers, and often remains neglected. Automating this task has the potential to save time while ensuring that messages are informative. A high-quality dataset and an objective benchmark are vital preconditions for solid research and evaluation towards this goal. We show that existing datasets exhibit various problems, such as the quality of the commit selection, small sample sizes, duplicates, privacy issues, and missing licenses for redistribution. This can lead to unusable models and skewed evaluations, where inferior models achieve higher evaluation scores due to biases in the data. We compile a new large-scale dataset, CommitBench, adopting best practices for dataset creation. We sample commits from diverse projects with licenses that permit redistribution and apply our filtering and dataset enhancements to improve the quality of generated commit messages. We use CommitBench to compare existing models and show that other approaches are outperformed by a Transformer model pretrained on source code. We hope to accelerate future research by publishing the source code( https://github.com/Maxscha/commitbench ).

翻译：编写提交信息是许多软件开发人员日常工作中繁琐的任务，且常被忽视。自动化该任务既能节省时间，又能确保提交信息具有信息量。高质量数据集和客观基准对于实现这一目标的可靠研究与评估至关重要。我们指出现有数据集存在多种问题，例如提交选择质量、样本量小、重复、隐私问题以及缺少允许重新分发的许可证。这些问题可能导致模型不可用与评估偏差——劣质模型因数据中的偏差反而获得更高的评估分数。我们采用数据集创建的最佳实践，整理出大规模新数据集CommitBench。我们从允许重新分发的许可协议项目中采样提交，并通过过滤与数据集增强手段提升生成提交信息的质量。利用CommitBench对现有模型进行比较，结果表明，基于源代码预训练的Transformer模型性能优于其他方法。我们通过公开源代码（https://github.com/Maxscha/commitbench）以期加速未来研究。