Data quality is crucial for training accurate, unbiased, and trustworthy machine learning models and their correct evaluation. Recent works, however, have shown that even popular datasets used to train and evaluate state-of-the-art models contain a non-negligible amount of erroneous annotations, bias or annotation artifacts. There exist best practices and guidelines regarding annotation projects. But to the best of our knowledge, no large-scale analysis has been performed as of yet on how quality management is actually conducted when creating natural language datasets and whether these recommendations are followed. Therefore, we first survey and summarize recommended quality management practices for dataset creation as described in the literature and provide suggestions on how to apply them. Then, we compile a corpus of 591 scientific publications introducing text datasets and annotate it for quality-related aspects, such as annotator management, agreement, adjudication or data validation. Using these annotations, we then analyze how quality management is conducted in practice. We find that a majority of the annotated publications apply good or very good quality management. However, we deem the effort of 30% of the works as only subpar. Our analysis also shows common errors, especially with using inter-annotator agreement and computing annotation error rates.

翻译：数据质量对于训练准确、无偏且可信的机器学习模型及其正确评估至关重要。然而近期研究表明，即便是用于训练和评估最先进模型的流行数据集，仍存在不可忽视的错误标注、偏差或标注伪影现象。尽管标注项目已有最佳实践指南，但据我们所知，目前尚无大规模研究分析自然语言数据集创建过程中实际实施的质量管理措施及其对推荐规范的遵循情况。为此，我们首先梳理文献中推荐的数据集创建质量管理实践并总结其应用建议，继而构建包含591篇文本数据集相关科学文献的语料库，对标注者管理、一致性检验、仲裁或数据验证等质量相关维度进行标注。基于这些标注信息，我们分析了实际质量管理实施状况。研究发现：多数标注文献采用良好或优秀的质量管理措施，但30%研究工作的质量管理效果仅达到次优水平。我们的分析还揭示了常见错误，特别是标注者间一致性检验与标注错误率计算中的问题。