Recently, deep learning models have been widely applied in program understanding tasks, and these models achieve state-of-the-art results on many benchmark datasets. A major challenge of deep learning for program understanding is that the effectiveness of these approaches depends on the quality of their datasets, and these datasets often contain noisy data samples. A typical kind of noise in program understanding datasets is label noise, which means that the target outputs for some inputs are incorrect. Researchers have proposed various approaches to alleviate the negative impact of noisy labels, and formed a new research topic: noisy label learning (NLL). In this paper, we conduct an empirical study on the effectiveness of noisy label learning on deep learning for program understanding datasets. We evaluate various NLL approaches and deep learning models on three tasks: program classification, vulnerability detection, and code summarization. From the evaluation results, we come to the following findings: 1) small trained-from-scratch models are prone to label noises in program understanding, while large pre-trained models are highly robust against them. 2) NLL approaches significantly improve the program classification accuracies for small models on noisy training sets, but they only slightly benefit large pre-trained models in classification accuracies. 3) NLL can effectively detect synthetic noises in program understanding, but struggle in detecting real-world noises. We believe our findings can provide insights on the abilities of NLL in program understanding, and shed light on future works in tackling noises in software engineering datasets. We have released our code at https://github.com/jacobwwh/noise_SE.

翻译：近年来，深度学习模型已广泛应用于程序理解任务，并在众多基准数据集上取得了最先进的结果。深度学习应用于程序理解的主要挑战在于，这些方法的效果高度依赖数据集质量，而数据集往往包含噪声样本。程序理解数据集中典型的一种噪声是标签噪声，即部分输入对应的目标输出是错误的。研究者提出了多种方法以减轻噪声标签的负面影响，并形成了新的研究方向：噪声标签学习。本文针对噪声标签学习在深度学习应用于程序理解数据集上的有效性开展实证研究。我们评估了多种噪声标签学习方法与深度学习模型在三个任务上的表现：程序分类、漏洞检测和代码摘要。从评估结果中得出以下发现：1）小规模从头训练的模型易受程序理解中的标签噪声影响，而大规模预训练模型对此具有较强的鲁棒性；2）噪声标签学习方法显著提升了小模型在含噪训练集上的程序分类准确率，但对大规模预训练模型在分类准确率上的提升有限；3）噪声标签学习能有效检测程序理解中的合成噪声，但在检测真实世界噪声时存在困难。我们相信这些发现可为噪声标签学习在程序理解中的能力提供见解，并为未来解决软件工程数据集噪声问题指明方向。相关代码已发布在 https://github.com/jacobwwh/noise_SE。