In industry deep learning application, our manually labeled data has a certain number of noisy data. To solve this problem and achieve more than 90 score in dev dataset, we present a simple method to find the noisy data and re-label the noisy data by human, given the model predictions as references in human labeling. In this paper, we illustrate our idea for a broad set of deep learning tasks, includes classification, sequence tagging, object detection, sequence generation, click-through rate prediction. The dev dataset evaluation results and human evaluation results verify our idea.

翻译：在工业级深度学习应用中，人工标注数据中存在一定数量的噪声数据。为解决该问题并在开发数据集上实现超过90分的成绩，我们提出了一种简洁方法：通过人工方式识别噪声数据并重新标注，其中利用模型预测结果作为人工标注的参考依据。本文阐述了该方法在广泛深度学习任务中的应用思路，涵盖分类、序列标注、目标检测、序列生成及点击率预测等任务。开发数据集评估结果与人工评估结果均验证了我们的设想。