Researchers recently found out that sometimes language models achieve high accuracy on benchmark data set, but they can not generalize very well with even little changes to the original data set. This is sometimes due to data artifacts, model is learning the spurious correlation between tokens and labels, instead of the semantics and logic. In this work, we analyzed SNLI data and visualized such spurious correlations. We proposed an adaptive up-sampling algorithm to correct the data artifacts, which is simple and effective, and does not need human edits or annotation. We did an experiment applying the algorithm to fix the data artifacts in SNLI data and the model trained with corrected data performed significantly better than the model trained with raw SNLI data, overall, as well as on the subset we corrected.

翻译：研究者近期发现，语言模型有时能在基准数据集上达到高准确率，但即使对原始数据集进行微小改动，其泛化能力也会显著下降。这有时源于数据伪影——模型学习的是词元与标签之间的虚假关联，而非语义与逻辑。本研究分析了SNLI数据并可视化此类虚假关联，提出一种无需人工编辑或标注的简单高效的自适应升采样算法以修正数据伪影。实验表明，将该算法应用于修正SNLI数据中的伪影后，使用修正数据训练的模型在整体性能及所修正子集上均显著优于原始SNLI数据训练的模型。