Given the prevalence of crowd sourced labor in creating Natural Language processing datasets, these aforementioned sets have become increasingly large. For instance, the SQUAD dataset currently sits at over 80,000 records. However, because the English language is rather repetitive in structure, the distribution of word frequencies in the SQUAD dataset's contexts are relatively unchanged. By measuring each sentences distance from the co-variate distance of frequencies of all sentences in the dataset, we identify 10,500 examples that create a more uniform distribution for training. While fine-tuning ELECTRA [4] on this subset of examples reaches better performance to a model trained on all 87,000 examples. Herein we introduce a methodology for systematically pruning datasets for fine tuning reaching better out of sample performance.

翻译：鉴于众包劳动力在自然语言处理数据集创建中的普遍性，上述数据集规模日益庞大。例如，SQUAD数据集当前已超过80,000条记录。然而，由于英语在结构上具有较高重复性，SQUAD数据集上下文中词频分布相对稳定。通过衡量每个句子与数据集中所有句子词频协变量距离的差异，我们识别出10,500个样本，可形成更均匀的训练分布。尽管在此子集上微调ELECTRA [4] 的性能优于基于全部87,000个样本训练的模型，本文提出了一种系统性修剪数据集以进行微调的方法，从而获得更优的样本外性能。