In industry deep learning application, our manually labeled data has a certain number of noisy data. To solve this problem and achieve more than 90 score in dev dataset, we present a simple method to find the noisy data and re-label the noisy data by human, given the model predictions as references in human labeling. In this paper, we illustrate our idea for a broad set of deep learning tasks, includes classification, sequence tagging, object detection, sequence generation, click-through rate prediction. The experimental results and human evaluation results verify our idea.

翻译：在工业深度学习应用中，人工标注的数据存在一定数量的噪声数据。为解决该问题并在开发数据集中达到90分以上的成绩，我们提出了一种简单方法，通过将模型预测结果作为人工标注的参考，发现噪声数据并由人工重新标注。本文阐述了该方法在广泛深度学习任务中的应用思路，涵盖分类、序列标注、目标检测、序列生成及点击率预测等任务。实验结果与人工评估结果验证了我们的设想。