In industry deep learning application, our manually labeled data has a certain number of noisy data. To solve this problem and achieve more than 90 score in dev dataset, we present a simple method to find the noisy data and re-label the noisy data by human, given the model predictions as references in human labeling. In this paper, we illustrate our idea for a broad set of deep learning tasks, includes classification, sequence tagging, object detection, sequence generation, click-through rate prediction. The experimental results and human evaluation results verify our idea.

翻译：在工业级深度学习应用中，人工标注的数据集往往包含一定数量的噪声数据。为解决该问题并实现在开发数据集上超过90分的成绩，我们提出一种简洁的方法：通过模型预测结果作为人工标注的参考，识别噪声数据并对其进行人工重标注。本文阐述的理念适用于广泛的深度学习任务，包括分类、序列标注、目标检测、序列生成和点击率预测。实验结果与人工评估验证了该方法的有效性。