Increasingly larger datasets have become a standard ingredient to advancing the state of the art in NLP. However, data quality might have already become the bottleneck to unlock further gains. Given the diversity and the sizes of modern datasets, standard data filtering is not straight-forward to apply, because of the multifacetedness of the harmful data and elusiveness of filtering rules that would generalize across multiple tasks. We study the fitness of task-agnostic self-influence scores of training examples for data cleaning, analyze their efficacy in capturing naturally occurring outliers, and investigate to what extent self-influence based data cleaning can improve downstream performance in machine translation, question answering and text classification, building up on recent approaches to self-influence calculation and automated curriculum learning.

翻译：日益庞大的数据集已成为推动自然语言处理技术发展的标准要素。然而，数据质量可能已成为制约性能提升的关键瓶颈。鉴于现代数据集的多样性与规模，由于有害数据的多面性以及难以跨任务泛化的过滤规则，标准数据过滤方法难以直接应用。本研究基于近期自影响计算与自动课程学习相关方法，探讨训练样本的任务无关自影响分数在数据清洗中的适用性，分析其捕获自然异常值的能力，并系统研究基于自影响的数据清洗在机器翻译、问答系统和文本分类任务中提升下游性能的潜力。